Re: Kernel docs: muddying the waters a bit
Jonathan Corbetwrites: > Indeed, I doubt many people want the DocBook itself. Might be nice to actually have a set of requirements before anyone tries to select a suitable system then :-) Here's my current set: asciidocsphinx htmlvia docbook native native (kinda) pdf via docbook via rst2pdf epubvia docbook?native man ? native? -- -keith signature.asc Description: PGP signature
Re: Kernel docs: muddying the waters a bit
On Fri, 04 Mar 2016, Russel Winderwrote: > On Thu, 2016-03-03 at 15:23 -0800, Keith Packard wrote: >> 1) the python version (asciidoc) appears to have been abandoned in >> favor of the ruby version. > > This is I think true, however the Java-based tool chain Asciidoctor is > I believe the standard bearer for ASCIIdoc these days, albeit called > ASCIIdoctor. If we're talking about the same asciidoctor (http://asciidoctor.org/) it's written in ruby but you can apparently run it in JVM using JRuby. Calling it Java-based is misleading. BR, Jani. -- Jani Nikula, Intel Open Source Technology Center -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
On Wed, May 4, 2016 at 3:43 PM, Daniel Vetterwrote: > I'd really like to converge on the markup question, so that we can start > using all the cool stuff with impunity in gpu documentations. Aside: If we decide this now I could send in a pull request for the rst/sphinx kernel-doc support still for 4.7 (based upon the minimal markdown/asciidoc code I still have). That would be really awesome ... Jon? Thanks, Daniel -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
On Wed, 04 May 2016, Jonathan Corbetwrote: > The sphinx/rst approach does seem, to me, to be the right one, with the > existing DocBook structure remaining in place for those who want/need > it. I'm inclined toward my stuff as a base to work with, obviously :) But > it's hackish at best and needs a lot of cleaning up. It's a proof of > concept, but it's hardly finished (one might say it's barely begun...) Thanks. I'll start looking at how to make it less hackish and more upstreamable. BR, Jani. -- Jani Nikula, Intel Open Source Technology Center -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
On Sat, 13 Feb 2016, Jonathan Corbet <cor...@lwn.net> wrote: > So can we discuss? I'm not saying we have to use Sphinx, but, should we > choose not to, we should do so with open eyes and good reasons for the > course we do take. What do you all think? This stalled a bit, but the waters are still muddy... Is the Sphinx/reStructuredText table support adequate for media/v4l documentation? Are the Sphinx output formats adequate in general? Specifically, is the lack of DocBook support, and the flexibility it provides, a blocker? Otherwise, I think Sphinx is promising. Jon, I think we need a roll of dice, err, a well-thought-out decision from the maintainer to go with one or the other, so we can make some real progress. BR, Jani. -- Jani Nikula, Intel Open Source Technology Center -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
On Tue, 16 Feb 2016 10:25:49 +0200 Jani Nikulawrote: > However I didn't think Sphinx could produce docbook, and a quick search > doesn't convince me otherwise. Do you have some links to back this up? Somehow I was really sure of it, but I'm not finding it now. There is an extension out there, but it warns about being a "work in progress," so I'm not sure we can count it. Whether this is a show-stopper is indeed a good question. I doubt many people wanted the DocBook for its own sake, it's a matter of where you can go from there. But yes, it would be good to be sure on this point. > Sphinx might offer a way to drop docproc through the extension > mechanism, without resorting to the "separate-file approach". It might > be a more sensible approach as a whole. There's a certain elegance to it that I like, but it is an idea that needs to actually be demonstrated. It could also come later on, though, with the docproc or include mechanisms used for now. Thanks, jon -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
On Tue, 16 Feb 2016, Jonathan Corbetwrote: > Whether this is a show-stopper is indeed a good question. I doubt many > people wanted the DocBook for its own sake, it's a matter of where you > can go from there. But yes, it would be good to be sure on this point. So the question is, are HTML, latex (for pdf and presumably ps), epub, texinfo, and man pages enough? My subjective opinion is they are enough, and whoever wants more can make it work for them using pandoc. It's not like we had this great publishing framework before that we'd be breaking. Quite the opposite, and it would be awesome to have a robust set of outputs actually working for the majority of people. > There's a certain elegance to it that I like, but it is an idea that > needs to actually be demonstrated. It could also come later on, though, > with the docproc or include mechanisms used for now. Oh, totally agreed, I was just musing on potential follow-up work. BR, Jani. -- Jani Nikula, Intel Open Source Technology Center -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
On Fri, 2016-03-04 at 09:46 +0200, Jani Nikula wrote: > […] > If we're talking about the same asciidoctor (http://asciidoctor.org/) > it's written in ruby but you can apparently run it in JVM using > JRuby. Calling it Java-based is misleading. Indeed, I was somewhat imprecise. Thanks to the work mostly of Charles Nutter, JRuby is invariably a faster platform for Ruby code than Ruby is. So yes ASCIIDoctor is JVM-based via JRuby, not Java-based. The real point here is that in a move from DocBook/XML as a documentation source, ASCIIDoctor is an excellent choice. -- Russel.=Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder@ekiga.net41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@winder.org.ukLondon SW11 1EN, UK w: www.russel.org.uk skype: russel_winder signature.asc Description: This is a digitally signed message part
Re: Kernel docs: muddying the waters a bit
Em Tue, 08 Mar 2016 05:13:13 -0700 Dan Allenescreveu: > On Tue, Mar 8, 2016 at 4:29 AM, Mauro Carvalho Chehab < > mche...@osg.samsung.com> wrote: > > > pandoc did a really crap job on the conversion. To convert this > > into something useful, we'll need to spend a lot of time, as it lost > > most of the cross-references, as they were defined via DocBook macros. > > > > I agree pandoc creates crappy AsciiDoc. We have a much better converter in > the works called DocBookRx. > > https://github.com/opendevise/docbookrx > > It has converted several very serious DocBook documents and we're > continuing to improve it. It's also a lot easier to hack than pandoc. Didn't work: $ ./bin/docbookrx ~/devel/docbook_test/v4l2.xml No visitor defined for ! Skipping. No visitor defined for ! Skipping. No visitor defined for ! Skipping. No visitor defined for ! Skipping. No visitor defined for ! Skipping. > > -Dan > > -- Thanks, Mauro -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
Mauro Carvalho Chehabwrites: > On my tests, Sphinix seemed too limited to format tables. Asciidoc > produced an output that worked better. Yes, asciidoc has much more flexibility in table formatting, including the ability to control text layout within cells and full control over borders. However, I think asciidoc has two serious problems: 1) the python version (asciidoc) appears to have been abandoned in favor of the ruby version. 2) It really is just a docbook pre-processor. Native html/latex output is poorly supported at best, and exposes only a small subset of the full capabilities of the input language. As such, we would have to commit to using the ruby version and either committing to fixing the native html output backend or continuing to use the rest of the docbook toolchain. We could insist on using the python version, of course. I spent a bit of time hacking that up to add 'real' support for a table-of-contents in the native HTML backend and it looks like getting those changes upstreamed would be reasonably straightforward. However, we'd end up 'owning' the code, and I'm not sure we want to. -- -keith signature.asc Description: PGP signature
Re: Kernel docs: muddying the waters a bit
> DocBook is a means to an end; nobody really wants DocBook itself as far > as I can tell. We only have docbook because it was the tool of choice rather a lot of years ago to then get useful output formats. It was just inherited when borrowed the original scripts from Gnome/Gtk. It's still the most effective way IMHO of building big structured documents out of the kernel. The Gtk people long ago rewrote the original document script into a real tool so they have some different and maintained tools that are close to equivalent and already have some markdown support. Before we go off and re-invent the wheel it might be worth just borrowing their wheel and tweaking it as needed ? In particular they can generate help indexes so that the entire output becomes nicely browsable with an HTML based help browser. Alan -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
On Thu, 03 Mar 2016 16:03:14 +0200 Jani Nikula <jani.nik...@intel.com> wrote: > This stalled a bit, but the waters are still muddy... I've been dealing with real-world obnoxiousness, something which won't come to an immediate end, unfortunately. But I have been taking some time to mess with things, and hope to have some more POC patches to send out soon. > Is the Sphinx/reStructuredText table support adequate for media/v4l > documentation? That's perhaps the biggest question. My sense is "yes", but this needs a bit more assurance than that. > Are the Sphinx output formats adequate in general? Specifically, is the > lack of DocBook support, and the flexibility it provides, a blocker? DocBook is a means to an end; nobody really wants DocBook itself as far as I can tell. I've been messing with rst2pdf a bit; it's not hard to get reasonable output, and, with some effort, we could probably get really nice output. HTML and EPUB are easily covered, still haven't really played around with man pages yet. And there's LaTeX if we really need it. I kind of think we're covered there, unless I've missed something? > Otherwise, I think Sphinx is promising. > > Jon, I think we need a roll of dice, err, a well-thought-out decision > from the maintainer to go with one or the other, so we can make some > real progress. My inclination at the moment is very much in the Sphinx direction. I had some vague thoughts of pushing a throwaway experimental directory with a couple of docs for 4.6 that would just let people play with it easily; then we'd see how many screams we get. We'll see if the world lets me get there. Thanks, jon -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
On Wed, Feb 17, 2016 at 11:14 PM, Jonathan Corbetwrote: > On Sun, 14 Feb 2016 13:27:04 +0100 > Daniel Vetter wrote: > >> One concern/open I have for pro/cons are the hyperlinks from kerneldoc >> comments. Currently we have the postproc hack, iirc Jani's patches >> generated links native when extracting the kerneldoc. What's the >> solution with spinx? > > So I've been trying to figure out what this refers to. Is this the > cross-reference links within the document? When I did my sphinx hack it > used a technique that, shall we say, strongly resembles what Jani's > patches did. One difference is that Sphinx has the concept of > "functions" built into it, so I use function references for those. That's what I meant. As long as I can type in stuff like func(), and similar and get a link for it automatically (plus anywhere else in the templated stuff for function headers) I'm really happy. -Daniel -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
On Tue, 08 Mar 2016, Dan Allenwrote: > One of the key goals of the Asciidoctor project is to be able to directly > produce a wide variety of outputs from the same source (without DocBook). > We've added flexibility and best practices into the syntax and matured the > converter mechanism to bridge this (sometimes very wide) gap. I think our conclusion so far was that the native AsciiDoc (and Asciidoctor) outputs fell short of our needs, forcing us to use the DocBook pipeline. I, for one, was hoping we could eventually simplify the toolchain. For example, there was no support for chunked, or split to chapters, HTML, and the single page result was simply way too big. > Asciidoctor is the future of AsciiDoc. Even the AsciiDoc Python maintainers > acknowledge that (including the original creator). Thanks for the input. We've touched the topic of AsciiDoc vs. Asciidoctor before [1]. So we should be using Asciidoctor instead of AsciiDoc. That actually makes choosing asciidoc harder, because requiring another language environment complicates, not simplifies, the toolchain. I'd really like to lower the bar for building the documentation, for everyone, so much so that it becomes part of the normal checks for patch inclusion. BR, Jani. [1] http://mid.gmane.org/86pow31ddj@hiro.keithp.com -- Jani Nikula, Intel Open Source Technology Center -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
Em Tue, 08 Mar 2016 05:09:40 -0700 Dan Allenescreveu: > Jani wrote: > > > there was no support for chunked, or split > > to chapters, HTML, and the single page result was simply way too big. > > > > That's not entirely true. First, you can pre-split at the source level > using includes and generate output for each of the masters. That's what I > tend to do and it works really well since these are logical split points. The problem on pre-splitting the documents and process them in separate is that this will break cross-references. At the media uAPI Docbook, we use a lot of cross references. Btw, we use a lot of includes. Currently, it has 187 separate files. We even parse the header files looking for typedefs, structs, enums, #defines and functions, in order to produce a document that will cross-reference the documentation. > Second, there is a custom converter in the works to split post-generate > (which is really what we're talking about when we compare it to the DocBook > toolchain). > > https://github.com/asciidoctor/asciidoctor-extensions-lab/blob/master/lib/multipage-html5-converter.rb > > It's just a prototype, but proves it is possible by design. I didn't test it, but I saw some comments at the web that the part that would handle cross-references between files is not ready. > Personally, I don't like most chunked HTML approaches because they split > arbitrarily. We are trying to find the right balance so that the output is > actually sensible. There's still work to do, but there are options in the > meantime. Well, if it is capable of creating one chunk per include file, and do cross-references between chunks, this would work for media UAPI book. Yet, it would be good to have the multi-chunk extension packaged on major distros, as I don't like the idea of installing it without using my distro's package manager. -- Thanks, Mauro -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
On Tue, 08 Mar 2016, Dan Allenwrote: > That's not entirely true. First, you can pre-split at the source level > using includes and generate output for each of the masters. That's what I > tend to do and it works really well since these are logical split points. I need to look into this again. Is there a specific option or directive to produce split output for includes? When I tried this, the result was just one big output file. (And indeed we'd need both. Some includes we want embedded, some includes should produce separate outputs.) >> That actually makes choosing asciidoc harder, because >> requiring another language environment complicates, not simplifies, the >> toolchain. I'd really like to lower the bar for building the >> documentation, for everyone, so much so that it becomes part of the >> normal checks for patch inclusion. > > Pardon my bluntness here, but I don't buy that argument. This is Linux. > Installing software couldn't be simpler, and we're talking about an > extremely well supported language (Ruby). Granted, that part works for me. I'm not so sensitive to the dependencies; others may disagree. > I think it's a huge exaggeration to say that Asciidoctor is any harder to > install than AsciiDoc Python. It's also a heck of a lot smaller in size > since AsciiDoc Python pulls in hundreds of MB of LaTeX packages. For me, the comparison is really between Sphinx and Asciidoctor, not so much doc vs. doctor. The native output format and extension support in Sphinx is appealing; I am not yet convinced we could manage with Asciidoctor but without DocBook. The extension offering seems better in Sphinx. > Whatever you decide, I wish you all the best with your documentation > efforts! Thanks! BR, Jani. -- Jani Nikula, Intel Open Source Technology Center -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
Em Tue, 8 Mar 2016 10:39:22 -0300 Mauro Carvalho Chehabescreveu: > Em Tue, 08 Mar 2016 05:13:13 -0700 > Dan Allen escreveu: > > > On Tue, Mar 8, 2016 at 4:29 AM, Mauro Carvalho Chehab < > > mche...@osg.samsung.com> wrote: > > > > > pandoc did a really crap job on the conversion. To convert this > > > into something useful, we'll need to spend a lot of time, as it lost > > > most of the cross-references, as they were defined via DocBook macros. > > > > > > > I agree pandoc creates crappy AsciiDoc. We have a much better converter in > > the works called DocBookRx. > > > > https://github.com/opendevise/docbookrx > > > > It has converted several very serious DocBook documents and we're > > continuing to improve it. It's also a lot easier to hack than pandoc. > > Didn't work: > > $ ./bin/docbookrx ~/devel/docbook_test/v4l2.xml > No visitor defined for ! Skipping. > No visitor defined for ! Skipping. > No visitor defined for ! Skipping. > No visitor defined for ! Skipping. > No visitor defined for ! Skipping. I tried to use docbookrx for the bits that were not properly converted, like the manpage-like pages: $ ../docbookrx/bin/docbookrx Documentation/DocBook/media/v4l/func-ioctl.xml No visitor defined for ! Skipping. Dan, if you want to take a look on what's going wrong here, the XML I'm trying to convert is: https://git.linuxtv.org/media_tree.git/tree/Documentation/DocBook/media/v4l/func-ioctl.xml If this would work, it should be generating something like: https://git.linuxtv.org/mchehab/asciidoc-poc.git/tree/func-ioctl.adoc Pandoc failed to fully convert it, but at least it left all the texts, with prevented rewriting it from scratch. This is the manual fix I applied to it: https://git.linuxtv.org/mchehab/asciidoc-poc.git/commit/func-ioctl.adoc?id=801d336c3742f26731e08c284290c32c0b4632fc FYI, we have 133 xml files at the media uAPI doc with refmeta. -- Thanks, Mauro -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
On Fri, Mar 04, 2016 at 10:29:08AM +0200, Jani Nikula wrote: > On Fri, 04 Mar 2016, Mauro Carvalho Chehabwrote: > > > > If, on the other hand, we decide to use RST, we'll very likely need to > > patch it to fulfill our needs in order to add proper table support. > > I've no idea how easy/difficult would be to do that, nor if Sphinx > > upstream would accept such changes. > > > > So, at the end of the day, we may end by having to carry on our own > > version of Sphinx inside our tree, with doesn't sound good, specially > > since it is not just a script, but a package with hundreds of > > files. > > If we end up having to modify Sphinx, it has a powerful extension > mechanism for this. We wouldn't have to worry about getting it merged to > Sphinx upstream, and we wouldn't have to carry a local version of all of > Sphinx. (In fact, the extension mechanism provides a future path for > doing kernel-doc within Sphinx instead of as a preprocessing step.) > > I know none of this alleviates your concerns with table supports right > now. I'll try to have a look at that a bit more. FWIW, I think table formatting in Sphinx works via style sheets. The mechanism is documented in the Python docutils docs that Sphinx is built upon. Basically you use the "class" or "role" directive and define the corresponding CSS or LaTeX (or rst2pdf) style. Here is one example (using a custom "cssclass" role): https://pythonhosted.org/sphinxjp.themes.basicstrap/sample.html Directives (especially role and class): http://www.sphinx-doc.org/en/stable/rest.html#directives LaTeX styling: http://docutils.readthedocs.org/en/sphinx-docs/user/latex.html#custom-interpreted-text-roles HTH, Johannes -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
Em Fri, 04 Mar 2016 15:09:09 +0100 Johannes Stezenbachescreveu: > On Fri, Mar 04, 2016 at 09:59:50AM -0300, Mauro Carvalho Chehab wrote: > > > > 3) I tried to use a .. cssclass, as Johannes suggested, but > > I was not able to include the CSS file. I suspect that this is > > easy to fix, but I want to see if the cssclass will also work for > > the pdf output as well. > > "cssclass" was (I think) a custom role defined in the example, > unless you also have defined a custom role you can use plain "class". > I have not looked deeper into the theming and template stuff. Well, it accepted cssclass for html (well, it didn't find the templates - so I guess it is just me failing to understand how to tell sphinx to get the stylesheet), but it rejects it for latexPDF. > > > 4) It seems that it can't produce nested tables in pdf: > > > > Markup is unsupported in LaTeX: > > v4l-table-within-table:: nested tables are not yet implemented. > > Makefile:115: recipe for target 'latexpdf' failed > > This: > http://www.sphinx-doc.org/en/stable/markup/misc.html#tables > > suggests you need to add the tabularcolumns directive > for complex tables. > > BTW, as an alternative to the ASCII-art input > there is also support for CSV and list tables: > http://docutils.sourceforge.net/docs/ref/rst/directives.html#table I converted one of the big tables to CSV. At least now it recognized it as a table. Yet, the table was very badly formated: https://mchehab.fedorapeople.org/media-kabi-docs-test/rst_tests/packed-rgb.html This is how this table should look like: https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html Also, as this table has merged cells at the legend. I've no idea how to tell sphinx to do that on csv format. The RST files are on this git tree: https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/ Regards, Mauro -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
On Fri, 06 May 2016, Markus Heiserwrote: > Am 06.05.2016 um 17:06 schrieb Jani Nikula : > >> On Fri, 06 May 2016, Markus Heiser wrote: >>> @Jonathan: what do you think? Should I prepare a patch >>> with a basic reST (sphinx) build infrastructure, including >>> >>> * a folder for sphinx docs: >>> >>> ./Documentation/sphinx/ >> >> I'm already working on a patch series taking a different approach. I >> don't think we should hide the documentation under an extra folder named >> after a tool. Actually, I'm strongly opposed to that. > > Could you post a link to a repo? / thanks Very much a work-in-progress https://cgit.freedesktop.org/~jani/drm/log/?h=sphinx I was hoping to polish it a bit more before showing it to the world. > There is no need for concurrency, let's work together on your repo. > Within my POC I realized similar building processes we will need in the > kernel sources ... where you have cascading configuration. A base > configuration which fits for all common cases and (if needed) a > *per-book* configuration. > > At the end, when it comes to generate pdf books/articles, man pages > and e.g. texinfo files out of a sphinx-project you will need a build > infrastructure like this. ... > You will need on sphinx-project for each DocBook and one single > sphinx-project where you collect the .txt to .rst migrated files. Surely you know more about Sphinx than I do, but I specifically would like to include e.g. gpu documentation in the main build. I'm really hoping we can have *additional* configuration files for special cases (only) as needed. BR, Jani. -- Jani Nikula, Intel Open Source Technology Center -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
On Wed, 4 May 2016 16:18:27 +0200 Daniel Vetterwrote: > > I'd really like to converge on the markup question, so that we can start > > using all the cool stuff with impunity in gpu documentations. > > Aside: If we decide this now I could send in a pull request for the > rst/sphinx kernel-doc support still for 4.7 (based upon the minimal > markdown/asciidoc code I still have). That would be really awesome ... Sorry for my relative absence...I'm still busy dealing with bureaucracy an ocean away from home. I hope to begin emerging from this mess in the near future. So ... there's the code you have, the work I (+Jani) did, and the work Markus has done. Which would you have me push into 4.7? The sphinx/rst approach does seem, to me, to be the right one, with the existing DocBook structure remaining in place for those who want/need it. I'm inclined toward my stuff as a base to work with, obviously :) But it's hackish at best and needs a lot of cleaning up. It's a proof of concept, but it's hardly finished (one might say it's barely begun...) In the end, I guess, I feel that anything we might try to push for 4.7 is going to look rushed and not ready, and Linus might react accordingly. I'd be more comfortable aiming for 4.8. I *will* have more time to focus on things in that time frame... I suspect you're pretty well fed up with this stuff being pushed back, and rightly so. All I can do is apologize. That said, if you do think there's something out there that is good enough to consider pushing in a week or two, do tell and we can all take a look. Thanks, jon -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
On Wed, 4 May 2016 14:57:38 -0300 Mauro Carvalho Chehabwrote: > Also, media documentation is not just one more documentation. It is > the biggest one we have, and that has more changes than any other > documentation under Documentation/DocBook: > > $ git lg --since 01/01/2015 ` ls *.tmpl|grep -v media`|wc -l > 116 > $ git lg --since 01/01/2015 ` ls *.tmpl|grep media` `find media/ -type f`|wc > -l > 179 > > It also is more than twice the size of the other DocBook docs: > > $ wc -l $(ls *.tmpl|grep media) `find media/ -type f`|tail -1 > 82275 total > $ wc -l $(ls *.tmpl|grep -v media)|tail -1 > 29568 total > > E. g. media corresponds to 60% of the number of patches and 73% of > the DocBook stuff. These numbers are not entirely representative, I have to say. You're ignoring the kerneldoc comments - which is what much of the "DocBook" documents are made of, and which is the focus of much of this activity. If you could find a way to count those, you'd get a different picture. But I don't think that really matters; there doesn't seem to be *that* much disagreement here. The media book is important; we want it to be a part of the overall kernel documentation suite and not stuck in some DocBook ghetto. I agree that we should have an idea for a plausible path for *all* of our documentation. But I'm also concerned about delaying this work yet again; we have developers trying to push forward with improved documentation, and they've had to wait a year for this stuff - so far. Thanks, jon -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
Hi Mauro, Am 04.05.2016 um 18:15 schrieb Mauro Carvalho Chehab: > Em Wed, 4 May 2016 11:34:08 +0200 > Markus Heiser escreveu: > >> Hi all, (hi Jonathan, please take note of my offer below) >> >> Am 03.05.2016 um 16:31 schrieb Daniel Vetter : >> >>> Hi all, >>> >>> So sounds like moving ahead with rst/sphinx is the option that should >>> allow us to address everyone's concerns eventually? Of course the >>> first one won't have it all (media seems really tricky), ... >> >> BTW: Mauro mentioned that ASCII-art tables are not diff-friendly ... >> For this, I wrote the "flat-table" reST-directive, which adds >> missing cells automatically: >> >> doc: >> http://return42.github.io/sphkerneldoc/articles/table_concerns.html#flat-table >> source: >> https://github.com/return42/sphkerneldoc/blob/master/doc/extensions/rstFlatTable.py > > Yeah, this should address the lack of a proper way to markup cell/row > spans, providing the additional bits for the tables we have at media. > > Yet, there are some issues with table conversions. See below. > Some tables, like the one here: > > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/control.html > > are truncated (tested with Mozilla and Chrome), and part of the information is > lost due to that. Not a problem of rendering. This was a bug in the magration from DocBook to reST. You might remember that we have discussed, that some of the tables are better marked-up as definition lists. This was (the last) one I forgot to convert to a definition list ... I hope it was the last one, if not and you find more or other broken parts, please inform me (on the linux-tv mailing, or direct). --Markus-- -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
On Tue, Apr 12, 2016 at 4:46 PM, Jonathan Corbetwrote: > On Fri, 8 Apr 2016 17:12:27 +0200 > Markus Heiser wrote: > >> motivated by this MT, I implemented a toolchain to migrate the kernel’s >> DocBook XML documentation to reST markup. >> >> It converts 99% of the docs well ... to gain an impression how >> kernel-docs could benefit from, visit my sphkerneldoc project page >> on github: >> >> http://return42.github.io/sphkerneldoc/ > > So I've obviously been pretty quiet on this recently. Apologies...I've > been dealing with an extended death-in-the-family experience, and there is > still a fair amount of cleanup to be done. > > Looking quickly at this work, it seems similar to the results I got. But > there's a lot of code there that came from somewhere? I'd put together a > fairly simple conversion using pandoc and a couple of short sed scripts; > is there a reason for a more complex solution? > > Thanks for looking into this, anyway; I hope to be able to focus more on > it shortly. Hi Jon, Thanks for digging into this. FWIW, here is my $0.02. I've been working on restarting the devicetree specification, and after looking at both reStructuredText and Asciidoc(tor) I thought I liked the Asciidoc markup better, so chose that. I then proceeded to spend weeks trying to get reasonable output from the toolchain. When I got fed up and gave Sphinx a try, I was up and running with reasonable PDF and HTML output in a day and a half. Honestly, in the end I think we could make either tool do what is needed of it. However, my impression after trying to do a document that needs to have nice publishable output with both tools is that Sphinx is easier to work with, simpler to extend, better supported. My vote is firmly behind Sphinx/reStructuredText. g. -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH linux v7 3/6] hwmon: occ: Add I2C transport implementation for SCOM operations
On Fri, 2017-02-10 at 15:05 -0600, Eddie James wrote: > On 02/09/2017 11:31 PM, Joel Stanley wrote: > > > On Wed, Feb 8, 2017 at 9:40 AM,<eaja...@linux.vnet.ibm.com> wrote: > > >> From: "Edward A. James"<eaja...@us.ibm.com> > >> > >> Add functions to send SCOM operations over I2C bus. The BMC can > >> communicate with the Power8 host processor over I2C, but needs to use SCOM > >> operations in order to access the OCC register space. > > This doesn't need to be separate from the p8_occ_i2c.c file. You can > > remove a layer of function calls by merging this in and having these > > be your getscom putscom bus_ops callbacks. > > The purpose of having this separate was so that we could do the scom > address shift for p8 separately. Separation was my suggestion. It makes the I2C transport implementation independent of any P8 details, and having it separate* is no more abstract than the core performing SCOMs through the FSI interface when that's available. I feel like it's muddying the waters conceptually to bury P8 details in the implementation of a SCOM transport layer. However, in our less abstract world the P8 will be the only system that uses the I2C transport, so while I don't think merging the functions is a good idea from an abstraction perspective it won't have a big impact. Andrew * Maybe the I2C SCOM transport even deserves to live outside drivers/hwmon/occ? signature.asc Description: This is a digitally signed message part
Re: [PATCH] doc: memory-barriers: reStructure Text
> Am 05.01.2018 um 04:52 schrieb afzal mohammed <afzal.mohd...@gmail.com>: [...] > Initially i was sceptical of rst & once instead of hitting the fly, > hit "make htmldocs" on the keyboard :), and the opinion about it was > changed. Yes, I understand that some of us have a (reasonable) doubt about the reST markup. It is not perfect in any matter, e.g. I don't like the ``monospace`` markup. But this is my home opinion. My hope is, that those of us who have a doubt give reST a chance ... it is a compromise, not as bad as you might think first ... your cooperation and your criticism is needed and welcome. Please let me invite you / read on: There are other plain-text markups e.g. AsciiDoc or Markdown. The reST markup and the Sphinx-builder is a compromise from a evaluation in 2016 (see linux-doc ML subject "muddying the waters" [1]). Jani wrote an article about the evaluation and it results [2]. And there are other articles documenting all the various aspects. - A report from the documentation maintainer [3] - Kernel documentation with Sphinx, part 2: how it works [4] - Kernel documentation update [5] To summarize it with my words: The old DocBook-based toolchain was hard to maintain and of course who want's to write XML? A consistent plain-text markup for articles in /Documentation/* and source-code comments (kernel-doc) was needed. The markup: IMO reST wins that race, because it has a extendable markup specification while others plain-text markups like Markdown have various (HTML) builders with various markup dialects[7] (which is more a mess than a definition). The builder: IMO Sphinx-Doc wins that race, since it is (well) maintained, widely used and has a interface for extensions. I.e. the one extension we wrote: 'kernel-doc' to parse kernel-doc comments from source code and include them in the articles. Perspective: Sphinx-Doc also offers solutions we might use in the future (e.g. building man-pages). Not to end in a mess, extensions should be implemented cautiously and deliberately (be patient). But that should not fool you; yes we have known problems with our toolchain and it is not yet ;) perfect in any matter (e.g. the highlighting in kernel-doc comments or the PDF generation or the sphinx-doc versions shipped with various distributions or ..) Anyway, today we have more than before: The reST learning curve is (compared to DocBook) not hard for newbees and our toolchain is flexible for all the requirements wich might come up in the future. IMO the actual challenge is the content and the organization of the doc-tree and for this [1] https://www.mail-archive.com/search?q=muddying+the+waters=linux-doc%40vger.kernel.org [2] https://lwn.net/Articles/692704/ [3] https://lwn.net/Articles/704613/ [4] https://lwn.net/Articles/692705/ [5] https://lwn.net/Articles/705224/ [6] http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html [7] http://pandoc.org/MANUAL.html#markdown-variants > It was easy to navigate through various docs & the realized > that various topics (& many) were present (yes, it was there earlier > also, but had to dive inside Documentation & search, while viewing the > toplevel index.html made them standout). It was like earlier you had > to go after docs, but now it was docs coming after you, that is my > opinion. > > Later while fighting with memory-barriers.txt, felt that it might be > good for it as well as to be in that company. > > And the readability as a text is not hurt as well. > > It was thought that rst conversion could be done quickly, but since > this was my first attempt with rst, had to put some effort to get a > not so bad output, even if this patch dies, i am happy to have learnt > rst conversion to some extent. Thats exactly what I mean: give reST a chance :) -- Markus -- -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
On Thu, 18 Feb 2016, Daniel Vetterwrote: > On Wed, Feb 17, 2016 at 11:14 PM, Jonathan Corbet wrote: >> On Sun, 14 Feb 2016 13:27:04 +0100 >> Daniel Vetter wrote: >> >>> One concern/open I have for pro/cons are the hyperlinks from kerneldoc >>> comments. Currently we have the postproc hack, iirc Jani's patches >>> generated links native when extracting the kerneldoc. What's the >>> solution with spinx? >> >> So I've been trying to figure out what this refers to. Is this the >> cross-reference links within the document? When I did my sphinx hack it >> used a technique that, shall we say, strongly resembles what Jani's >> patches did. One difference is that Sphinx has the concept of >> "functions" built into it, so I use function references for those. > > That's what I meant. As long as I can type in stuff like func(), > and similar and get a link for it automatically (plus anywhere > else in the templated stuff for function headers) I'm really happy. I think that could be made to work in rst just as well as asciidoc. Which is to say, kernel-doc may generate broken refs in both, since it doesn't know if the link target exists outside of the file. Also, in theory, it's possible to generate non-unique targets in the end result if there are same named enums, structs, static functions etc. but I think that's less of a problem. Asciidoc just ignores these issues, I don't know what sphinx does. I think some of that could be alleviated by making the kernel-doc inclusion a directive through a sphinx extension. It could at the very least provide informative error messages. But that's distant future. Worth noting is that, AFAICT, in all of the proposals, including the original where kernel-doc produces docbook, this autoreferencing only works within parts processed by kernel-doc. Not in the template documents themselves. (You can still use the markup's more verbose cross referencing keywords.) BR, Jani. -- Jani Nikula, Intel Open Source Technology Center -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
Em Tue, 8 Mar 2016 12:39:21 -0300 Mauro Carvalho Chehabescreveu: > Pandoc failed to fully convert it, but at least it left all the texts, > with prevented rewriting it from scratch. This is the manual fix > I applied to it: > > https://git.linuxtv.org/mchehab/asciidoc-poc.git/commit/func-ioctl.adoc?id=801d336c3742f26731e08c284290c32c0b4632fc > > FYI, we have 133 xml files at the media uAPI doc with refmeta. I used pandoc to convert from the html files and manually edited it. I also fixed lots of other issues with the conversion. I guess the conversion to asciidoc format is now in good shape, at least to demonstrate that it is possible to use this format for the media docbook. Still, there are lots of broken references. The proof of concept html file is at: https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/media_api.html I also added the ascii doc files there, at: https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/ And I'm keeping the git tree, with helps to identify the work that was needed to make it work: https://git.linuxtv.org/mchehab/asciidoc-poc.git In summary, AsciiDoc, formatted via AsciiDoctor worked fine to produce an html file. PROBLEMS 1) I was not able to produce outputs on any other format. For example, when trying to generate docbook45 output, it sems that part of the trouble was due to pandoc conversion. It produces links like: link:#ftn.id-1.4.11.43.5.11.2.7.2.6.2[^[a]^] Which causes errors with DocBook parsers, like xmllint: media_api.xml:32300: parser error : Opening and ending tag mismatch: superscript line 32300 and ulink [a] ^ I suspect that this is fixable. I may try to fix it later. 2) It seems that Asciidoctor doesn't allow annexes per document part. It numberates them as chapters, instead of using A, B, C, ... 3) Even producing the html without troubles, it produces an error: asciidoctor: ERROR: media_api.adoc: line 57: invalid part, must have at least one section (e.g., chapter, appendix, etc.) 4) There are some things that got lost during the conversion, like copyright notes and revision notes. This could be simply a problem with pandoc conversion. Nothing serious, I guess, as we could insert the lost data manually. Yet, it means that, to move from the PoC to the Kernel, there are still lots of work to do. I was unable do discover why, nor to suppress this error message. Yet, from my side, if we're willing to get rid of DocBook, then Asciidoctor seems to be the *only* alternative so far to parse the complex media documents. Regards, Mauro -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
Em Thu, 03 Mar 2016 15:23:23 -0800 Keith Packardescreveu: > Mauro Carvalho Chehab writes: > > > On my tests, Sphinix seemed too limited to format tables. Asciidoc > > produced an output that worked better. > > Yes, asciidoc has much more flexibility in table formatting, including > the ability to control text layout within cells and full control over > borders. > > However, I think asciidoc has two serious problems: > > 1) the python version (asciidoc) appears to have been abandoned in > favor of the ruby version. > > 2) It really is just a docbook pre-processor. Native html/latex output > is poorly supported at best, and exposes only a small subset of the > full capabilities of the input language. > > As such, we would have to commit to using the ruby version and either > committing to fixing the native html output backend or continuing to use > the rest of the docbook toolchain. > > We could insist on using the python version, of course. I spent a bit of > time hacking that up to add 'real' support for a table-of-contents in > the native HTML backend and it looks like getting those changes > upstreamed would be reasonably straightforward. However, we'd end up > 'owning' the code, and I'm not sure we want to. I'm a way more concerned about using a tool that fulfill our needs than to look for something that won't use the docbook toolchain or require to install ruby. In the case of Docbook, we know it works and we know already its issues. Please correct me if I'm wrong, but the big problem we have is not due to the DocBook toolchain, but due to the lack of features at the kernel-doc script. Also, xmlto is already installed by the ones that build the kernel docs. So, keeping use it won't require to install a weird toolchain by hand. So, to be frank, it doesn't scary me to use either pyhton or ruby script + docbook. Of course, having to own the code has a cost that should be evaluated. If, on the other hand, we decide to use RST, we'll very likely need to patch it to fulfill our needs in order to add proper table support. I've no idea how easy/difficult would be to do that, nor if Sphinx upstream would accept such changes. So, at the end of the day, we may end by having to carry on our own version of Sphinx inside our tree, with doesn't sound good, specially since it is not just a script, but a package with hundreds of files. Thanks, Mauro pgp9vItgvYAsJ.pgp Description: Assinatura digital OpenPGP
Re: Kernel docs: muddying the waters a bit
On Thu, 2016-03-03 at 15:23 -0800, Keith Packard wrote: > […] > However, I think asciidoc has two serious problems: > > 1) the python version (asciidoc) appears to have been abandoned in > favor of the ruby version. This is I think true, however the Java-based tool chain Asciidoctor is I believe the standard bearer for ASCIIdoc these days, albeit called ASCIIdoctor. > 2) It really is just a docbook pre-processor. Native html/latex > output > is poorly supported at best, and exposes only a small subset of > the > full capabilities of the input language. This is not true. Yes ASCIIDoc started as a DocBook/XML frontend so as to use a sane :-) markup language rather than XML (XML is a notation for consenting computers only), but the current ASCIIDoctor toolchain deals very well in direct HTML and PDF generation, without needing a DocBook/XML toolchain. > As such, we would have to commit to using the ruby version and either > committing to fixing the native html output backend or continuing to > use > the rest of the docbook toolchain. Or trial the JVM-based ASCIIdoctor which is what the projects I am involved with chose to use. Perhaps as an example I can give you http:/ /gpars.website (it's a redirector) all the HTML and PDF is generated from ASCIIDoc source using ASCIIDoctor driven with a Gradle build system. This is still very much a work in progress (by Jim Northrop, not me currently), but I like it. > We could insist on using the python version, of course. I spent a bit > of > time hacking that up to add 'real' support for a table-of-contents in > the native HTML backend and it looks like getting those changes > upstreamed would be reasonably straightforward. However, we'd end up > 'owning' the code, and I'm not sure we want to. If the Python version is really not being maintained, I would suggest that unless you want to take over the project and be it's maintainer, you would be better advised to use a different version. -- Russel.=Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder@ekiga.net41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@winder.org.ukLondon SW11 1EN, UK w: www.russel.org.uk skype: russel_winder signature.asc Description: This is a digitally signed message part
Re: Kernel docs: muddying the waters a bit
On Thu, 3 Mar 2016 14:34:25 + One Thousand Gnomeswrote: > We only have docbook because it was the tool of choice rather a lot of > years ago to then get useful output formats. It was just inherited when > borrowed the original scripts from Gnome/Gtk. It's still the most > effective way IMHO of building big structured documents out of the kernel. ...except that we haven't used it that way. Instead, we make a whole bunch of smaller, partially structured document silos. > The Gtk people long ago rewrote the original document script into a real > tool so they have some different and maintained tools that are close to > equivalent and already have some markdown support. Before we go off and > re-invent the wheel it might be worth just borrowing their wheel and > tweaking it as needed ? In particular they can generate help indexes so > that the entire output becomes nicely browsable with an HTML based help > browser. Well, not inventing the wheel was kind of the motivation behind much of this effort; I got kind of worried watching us trying to cobble more functionality into our existing house-of-cards documentation system. Sphinx is a well-established, heavily used, and well supported system; using it would not be an exercise in wheel reinvention. As far as I can tell, it does everything we need (with some open questions about table support), lets us drop the whole DocBook toolchain dependency, and move to a much better-supported setup than we have now. Plus we get much nicer output, index generation, cross-references between documents, and the ability to write documents in a lightweight markup language. Seems like a win. I assume you're referring to gtk-doc? It's web page (http://www.gtk.org/gtk-doc/) starts by noting that it's "a bit awkward to setup and use"; they recommend looking at Doxygen instead. So I guess I'm not really sure what it offers that merits throwing another option into the mix now? What am I missing? Thanks, jon -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
On Wed, 09 Mar 2016, Dan Allenwrote: > On Tue, Mar 8, 2016 at 6:58 AM, Jani Nikula wrote: > >> I need to look into this again. Is there a specific option or directive >> to produce split output for includes? When I tried this, the result was >> just one big output file. (And indeed we'd need both. Some includes we >> want embedded, some includes should produce separate outputs.) >> > > Nope. What I'm saying is that you run Asciidoctor on each sub-master > include file (an include that manages a part or chapter). That gives you > your individual part/chapter files. Then you need to make an index page, > probably by using the Asciidoctor API to itemize all the chapters as a list > or something. Bummer. Getting the inter-document cross references right may become tricky. We'll be generating plenty of snippets of lightweight markup from source code documentation comments. At the time of processing, we won't know where e.g. a specific function to be cross referenced is documented, if at all. We can't require the documentation comment writers to figure that out either; it's too burdensome, too ugly in the code, and they'll bitrot quickly. Cross referencing in the asciidoc proofs of concept have worked because they've all done the processing as a single single unit, with includes. These hacks have also ignored any broken links, and there have been > Yes, it does require some thinking about cross references. There is a lot > more we can do out of the box, but all those references can be fixed with a > little bit of post-processing in the meantime. It seems to me Sphinx provides much better support regarding cross references, out of the box, within documents and to external documents (intersphinx), with target roles and domains, including validation and not creating broken links in the output. Looking at the current hacks we have for post-processing references, I'm really not thrilled about the prospect of keeping or redoing that. See how this works in Jon's Sphinx test [1]. At the time of generating the markup from source comments, there is no idea if and where gem_init_hw() and intel_guc_ucode_init() are documented. Indeed, documentation for the former does not exist, but there's no broken link. BR, Jani. [1] http://static.lwn.net/kerneldoc/gpu.html#c.intel_guc_ucode_load -- Jani Nikula, Intel Open Source Technology Center -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
Em Fri, 6 May 2016 18:06:49 +0300 Jani Nikulaescreveu: > On Fri, 06 May 2016, Markus Heiser wrote: > > @Jonathan: what do you think? Should I prepare a patch > > with a basic reST (sphinx) build infrastructure, including > > > > * a folder for sphinx docs: > > > > ./Documentation/sphinx/ > > I'm already working on a patch series taking a different approach. I > don't think we should hide the documentation under an extra folder named > after a tool. Actually, I'm strongly opposed to that. > > Instead, we should place the Sphinx stuff directly under Documentation, > and have Sphinx recursively pick up all the *.rst files. We should > promote gradually switching to lightweight markup and integration of the > documents into one system. This process should be as little disruptive > as possible. We won't avoid the need of moving things among directories, as we have lots of stuff under DocBook/ dir (btw, named after the toolchain). Ok, if we put the .rst files at Documentation, we very likely reduce the number of renames, but we'll increase the Makefile complexity, and the risk of breakages. One alternative would be to put the sphinx stuff on a separate Makefile, but using multiple makefiles on a single dir is not standard at the Kernel. > If someone wants to convert a .txt document to .rst and get it processed > by Sphinx, it should be as simple as renaming the file, doing the > necessary edits, and adding it to a toctree. Imagine gradually > converting the files under, say, Documentation/kbuild. Why should the > .rst files be moved under another directory? They should stay alongside > the .txt files under the same directory. There's bound to be a lot of > people who'll never use Sphinx, and will expect to find the good old > plain text files (albeit with some markup) where they always were. Well, git will show the change as a rename, no matter if the directory name changes or not (except if we keep the rst files with .txt extension), but I agree with you that people will expect to see text files at Documentation, and most will just read it without caring to run Sphinx. -- Thanks, Mauro -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
On Wed, 04 May 2016, Markus Heiserwrote: > Correct my, if I'am wrong. I'am a bit unfamiliar with DOCPROC in > particular with your "MARKDOWNREADY := gpu.xml" process. > > As I understood, you convert markdown to docbook within the kernel-doc > script using pandoc's executable? ... I don't face this topic. With my > modification of kernel-doc I produced pure reST markup from standardize > kernel-doc markup, no intermediate steps or tools required. That pandoc thing is a dead end. Forget about it. I think we've all pretty much agreed we should have kernel-doc produce the lightweight markup directly since [1]. [1] http://mid.gmane.org/1453106477-21359-1-git-send-email-jani.nik...@intel.com > Am 04.05.2016 um 17:09 schrieb Jonathan Corbet : > >> I think all of this makes sense. It would be really nice to have the >> directives in the native sphinx language like that. I *don't* think we >> need to aim for that at the outset; the docproc approach works until we can >> properly get rid of it. What would be *really* nice would be to get >> support for the kernel-doc directive into the sphinx upstream. > > No need for kernel-doc directive in sphinx upstream, later it will be > an extension which could be installed by a simple command like > "pip install kernel-doc-extensions" or similar. > > I develop these required extension (and more) within my proof of concept > on github ... this takes time ... if I finished all my tests and all is > well, I will build the *kernel-doc-extensions* package and deploy it > on https://pypi.python.org/pypi from where everyone could install this > with "pip". I think we should go for vanilla sphinx at first, to make the setup step as easy as possible for everyone. Even if it means still doing that ugly docproc step to call kernel-doc. We can improve from there, and I definitely appreciate your work on making this work with sphinx extensions. That said, how would it work to include the kernel-doc extension in the kernel source tree? Having things just work if sphinx is installed is preferred over requiring installation of something extra from pypi. (I know this may sound backwards for a lot of projects, but for kernel I'm pretty sure this is how it should be done.) BR, Jani. -- Jani Nikula, Intel Open Source Technology Center -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
Em Wed, 4 May 2016 11:34:08 +0200 Markus Heiserescreveu: > Hi all, (hi Jonathan, please take note of my offer below) > > Am 03.05.2016 um 16:31 schrieb Daniel Vetter : > > > Hi all, > > > > So sounds like moving ahead with rst/sphinx is the option that should > > allow us to address everyone's concerns eventually? Of course the > > first one won't have it all (media seems really tricky), ... > > BTW: Mauro mentioned that ASCII-art tables are not diff-friendly ... > > Am 18.04.2016 um 13:16 schrieb Mauro Carvalho Chehab > : > > > With that sense, the "List tables" format is also not good, as > > one row addition would generate several hunks (one for each column > > of the table), making harder to review the patch by just looking at > > the diff. > > For this, I wrote the "flat-table" reST-directive, which adds > missing cells automatically: > > doc: > http://return42.github.io/sphkerneldoc/articles/table_concerns.html#flat-table > source: > https://github.com/return42/sphkerneldoc/blob/master/doc/extensions/rstFlatTable.py Yeah, this should address the lack of a proper way to markup cell/row spans, providing the additional bits for the tables we have at media. Yet, there are some issues with table conversions. See below. > > I used "flat-table" to migrate all DocBook-XML documents to reST. With this > directive, I also managed to migrate the complete media book (no more TODOs) > incl. the large tables like them from subdev-formats: > > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html > > (Rendering large tables is a general discussion which should not take place > in this MT) Some tables, like the one here: https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/control.html are truncated (tested with Mozilla and Chrome), and part of the information is lost due to that. Regards, Mauro -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
On Wed, May 04, 2016 at 08:57:13AM -0600, Jonathan Corbet wrote: > On Wed, 4 May 2016 16:18:27 +0200 > Daniel Vetterwrote: > > > > I'd really like to converge on the markup question, so that we can start > > > using all the cool stuff with impunity in gpu documentations. > > > > Aside: If we decide this now I could send in a pull request for the > > rst/sphinx kernel-doc support still for 4.7 (based upon the minimal > > markdown/asciidoc code I still have). That would be really awesome ... > > Sorry for my relative absence...I'm still busy dealing with bureaucracy > an ocean away from home. I hope to begin emerging from this mess in the > near future. > > So ... there's the code you have, the work I (+Jani) did, and the work > Markus has done. Which would you have me push into 4.7? > > The sphinx/rst approach does seem, to me, to be the right one, with the > existing DocBook structure remaining in place for those who want/need > it. I'm inclined toward my stuff as a base to work with, obviously :) But > it's hackish at best and needs a lot of cleaning up. It's a proof of > concept, but it's hardly finished (one might say it's barely begun...) > > In the end, I guess, I feel that anything we might try to push for 4.7 is > going to look rushed and not ready, and Linus might react accordingly. > I'd be more comfortable aiming for 4.8. I *will* have more time to focus > on things in that time frame... I suspect you're pretty well fed up with > this stuff being pushed back, and rightly so. All I can do is apologize. > > That said, if you do think there's something out there that is good > enough to consider pushing in a week or two, do tell and we can all take > a look. Well I'd just have taken the asciidoc hacks I have currently in my topic/kerneldoc branch, converted to sphinx and looked how it fares. It should be fairly minimal, and I think the first step we want to do for the long-term plan. I hope I can ready something, and then we can look whether it's rushed for 4.7 or not. Thanks, Daniel -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
Em Wed, 4 May 2016 08:57:13 -0600 Jonathan Corbetescreveu: > On Wed, 4 May 2016 16:18:27 +0200 > Daniel Vetter wrote: > > > > I'd really like to converge on the markup question, so that we can start > > > using all the cool stuff with impunity in gpu documentations. > > > > Aside: If we decide this now I could send in a pull request for the > > rst/sphinx kernel-doc support still for 4.7 (based upon the minimal > > markdown/asciidoc code I still have). That would be really awesome ... > > Sorry for my relative absence...I'm still busy dealing with bureaucracy > an ocean away from home. I hope to begin emerging from this mess in the > near future. > > So ... there's the code you have, the work I (+Jani) did, and the work > Markus has done. Which would you have me push into 4.7? IMHO, Markus approach is the one that is providing a better result, as it added support for missing features that we require for the media DocBook. Still, it seems premature to merge it for 4.7. Markus is getting real progress on converting the media docs to work with Sphynx, but there are still troubles when the table is big, as, currently, it is truncating everything that passes the right margin on some tables. So, IMHO, the next action item would be to be sure that big tables will not be truncated. Assuming that he can fix that in time, we can merge it for 4.8, and then start porting the documentation to use it. Btw, converting the DocBook/media Makefile will require an extra step, as part of the documentation is generated via scripts (some C file examples, and include/uapi headers). Those scripts also warrant that (almost) every API at include/uapi is synchronized with the DocBook. I use this on my patch review process, in order to reject patches that don't add the proper documentation updates. Thanks, Mauro -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: rst2pdf (was [PATCH 00/10] Documentation/Sphinx)
Am 03.06.2016 um 22:47 schrieb Jonathan Corbet <cor...@lwn.net>: > On Mon, 30 May 2016 23:05:34 +0300 > Jani Nikula <jani.nik...@intel.com> wrote: > >>> I can't recommend to use rst2pdf (it is less maintained), use default >>> sphinx LaTeX toolchain. >> >> I think we'll use whatever works, rst2pdf seemed to work for now, but we >> can change if needed. > > I really like the idea of using rst2pdf and keeping the huge latex > dependency out of the mix. I am a bit concerned, though; I've been able > to crash it in my experiments here. We may want to have the ability to > support either chain eventually; otherwise, we might just end up picking > up maintenance of rst2pdf at some point so that it works properly for us. > > jon I looked closer to rst2pdf, it supports only the docutils reST, but not the sphinx superset ... - $ rst2pdf index.rst index.rst:15: (ERROR/3) Unknown interpreted text role "ref". index.rst:15: (ERROR/3) Unknown interpreted text role "ref". index.rst:27: (ERROR/3) Unknown directive type "toctree". .. toctree:: :maxdepth: 1 kernel-doc-intro kernel-doc-syntax - rules like ":ref:", domains like ":c:type:" and directives like ".. toctree:" are a part of the (extended) reST syntax from sphinx, thats why standard docutils (like rst2*) will not work ... > Am 18.04.2016 um 10:10 schrieb Markus Heiser <markus.hei...@darmarit.de>: > Re: Kernel docs: muddying the waters a bit > > BTW a few words about differences between DockBook and reST (Sphinx). > > With DocBook you write *books*, the protocol (the DocBook application) has > no facility to *chunk* and interconnect several documents. The external > ENTITY > is a workaround on the SGML layer, not on XML nor on the DB-application layer. > Thats the reason, why so many XML-tools don't handle this entities and > many DocBook to (e.g.) reST tools are fail. > > With **standard** reST it is nearly the same, except there is a "include" > directive on the application layer. But this directive is very simple, > comparable to the C preprocessor "#include" directive. > > With the **superset** reST-markup of Sphinx-doc you get a the "toctree" > directive, > which lets you control how a document-tree should be build. > > http://www.sphinx-doc.org/en/stable/markup/toctree.html > > @Mauro: you mentioned a docutils (rst2*) experience in your mail > http://marc.info/?l=linux-doc=145735316012094=2 > > Because the "toctree" directive -- and other directives > we use -- are a part of a superset of the **standard** > reST, the standard docutils (like rst2*) will not work. -- Markus -- -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Kernel docs: muddying the waters a bit
So I fear you all are going to hate me for this... Asciidoc is a credible solution to the formatted documentation problem, but it's not the only such; I'd like to be sure that we pick the right one. I worry that asciidoc seems to be aimed mostly at small documents, and that the project itself seems a little lifeless - it's not a good sign when your main page's link to the repository has been dead for a long time. (Asciidoctor seems more active, with the Github folks behind it, but that means bringing Ruby into the picture). An alternative we haven't really looked at yet is ReStructuredText (or "RST") and the Sphinx system (sphinx-doc.org) built on top of it. RST is YA simple markup scheme, remarkably similar to Markdown or Asciidoc; Sphinx is a fairly sophisticated documentation system that uses RST. I spent a few hours reworking the asciidoc patches to do RST instead, then built a few template files' worth of docs. The result can be seen at: http://static.lwn.net/kerneldoc/ It's very much a POC (however you might want to define the term), there's lots of glitches, I chose a theme pretty much at random, etc. But it shows that it can be done. Like asciidoc, Sphinx is Python-based, so it adds little to the toolchain requirements there. It produces integrated, multi-file HTML natively, with a TOC, an index, cross-file cross references, and more. It can make things like function indexes. It claims output in epub, docbook, and man (I've not yet messed with those). The path to PDF is via latex; clearly the docbook path could be used too. I used my same docproc hack to extract the comments here, mostly because I had it at hand. We could go with Jani's separate-file approach if we wanted. There's also a tool out there (called "breathe") that's meant to turn doxygen-style comments into RST; I haven't had a chance to mess with it. We *could* also write an extension to pull the comments directly in Sphinx if there were a compelling reason to do so. If anybody's curious, the work done to get this far is in: git://git.lwn.net/linux.git doc/sphinx but it looks suspiciously like the previous asciidoc patches, and, in any case, it would have to be thrown out, publicly disowned, and replaced before going any further with this, should that be what we decide to do. So can we discuss? I'm not saying we have to use Sphinx, but, should we choose not to, we should do so with open eyes and good reasons for the course we do take. What do you all think? jon -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
On Thu, Feb 18, 2016 at 10:11 AM, Jani Nikulawrote: > On Thu, 18 Feb 2016, Daniel Vetter wrote: >> On Wed, Feb 17, 2016 at 11:14 PM, Jonathan Corbet wrote: >>> On Sun, 14 Feb 2016 13:27:04 +0100 >>> Daniel Vetter wrote: >>> One concern/open I have for pro/cons are the hyperlinks from kerneldoc comments. Currently we have the postproc hack, iirc Jani's patches generated links native when extracting the kerneldoc. What's the solution with spinx? >>> >>> So I've been trying to figure out what this refers to. Is this the >>> cross-reference links within the document? When I did my sphinx hack it >>> used a technique that, shall we say, strongly resembles what Jani's >>> patches did. One difference is that Sphinx has the concept of >>> "functions" built into it, so I use function references for those. >> >> That's what I meant. As long as I can type in stuff like func(), >> and similar and get a link for it automatically (plus anywhere >> else in the templated stuff for function headers) I'm really happy. > > I think that could be made to work in rst just as well as > asciidoc. Which is to say, kernel-doc may generate broken refs in both, > since it doesn't know if the link target exists outside of the > file. Also, in theory, it's possible to generate non-unique targets in > the end result if there are same named enums, structs, static functions > etc. but I think that's less of a problem. Asciidoc just ignores these > issues, I don't know what sphinx does. > > I think some of that could be alleviated by making the kernel-doc > inclusion a directive through a sphinx extension. It could at the very > least provide informative error messages. But that's distant future. > > Worth noting is that, AFAICT, in all of the proposals, including the > original where kernel-doc produces docbook, this autoreferencing only > works within parts processed by kernel-doc. Not in the template > documents themselves. (You can still use the markup's more verbose cross > referencing keywords.) It works everywhere, even in the docbook template, as long as you mark it up correctly. Which in docbook means func. That's because it's a post-proc path over the entire doc. But then the entire point here is to move the overview sections all into kerneldoc, so making the links in the templates more verbose shouldn't be a big deal. -Daniel -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
Em Mon, 7 Mar 2016 09:48:26 +0100 Johannes Stezenbachescreveu: > On Mon, Mar 07, 2016 at 12:29:08AM +0100, Johannes Stezenbach wrote: > > On Sat, Mar 05, 2016 at 11:29:37PM -0300, Mauro Carvalho Chehab wrote: > > > > > > I converted one of the big tables to CSV. At least now it recognized > > > it as a table. Yet, the table was very badly formated: > > > > > > https://mchehab.fedorapeople.org/media-kabi-docs-test/rst_tests/packed-rgb.html > > > > > > This is how this table should look like: > > > https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html > > > > > > Also, as this table has merged cells at the legend. I've no idea how > > > to tell sphinx to do that on csv format. > > > > > > The RST files are on this git tree: > > > https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/ > > > > Yeah, seems it can't do merged cells in csv. Attached patch converts it > > back to grid table format and fixes the table definition. > > The html output looks usable, but clearly it is no fun to > > work with tables in Sphinx. > > > > Sphinx' latex writer can't handle nested tables, though. > > Python's docutils rst2latex can, but that doesn't help here. > > rst2pdf also supports it. But I have doubts such a large > > table would render OK in pdf without using landscape orientation. > > I have not tried because I used python3-sphinx but rst2pdf > > is only availble for Python2 in Debian so it does not integrate > > with Sphinx. > > Just a quick idea: > Perhaps one alternative would be to use Graphviz to render > the problematic tables, it supports a HTML-like syntax > and can be embedded in Spinx documents: > > http://www.sphinx-doc.org/en/stable/ext/graphviz.html > http://www.graphviz.org/content/node-shapes#html > http://stackoverflow.com/questions/13890568/graphviz-html-nested-tables That could work, but it is scary... Graphviz is great to generate diagrams, but it really sucks when one wants to put a graph element on a specific place, as it loves to reorder elements putting them on unexpected places. Btw, I converted all docs from our uAPI docbook to rst using pandoc. It was a brainless conversion, except for a few fixes. The output is at: https://mchehab.fedorapeople.org/media-kabi-docs-test/rst_tests/ I added it on the top of my PoC tree at: https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/ Besides tables, I noticed some other bad things that needs to be corrected somehow: 1) Document divisions are not numbered. We need that. It should be broken into: - Document divisions - one per documented API: - V4L2 - Remote Controllers - DVB - Media Controller - Chapters - Sessions Everything should be numbered, as, when discussing API improvements, it is usual the need of pinpoint to an specific chapter and section. Tables and images should also be numbered, and we need a way to use references for table/image numbers. 2) Images Most images didn't popup. We have images on different file formats: - SVG - GIF - PDF - PNG 3) References It could be a conversion issue, but there are lots of missing references at the documentation. 4) We need to have some way to tell sphinx to not put some things at the lateral ToC bar. For example, at the V4L2 "Changes" section, we don't want to have one entry per version at the ToC bar. Giving that, I suspect that we'll have huge headaches to address if we use sphinx, as it seems too limited to handle complex documents. We should try to use some other tool that would give us better results. Regards, Mauro -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
Em Mon, 7 Mar 2016 00:29:08 +0100 Johannes Stezenbachescreveu: > On Sat, Mar 05, 2016 at 11:29:37PM -0300, Mauro Carvalho Chehab wrote: > > > > I converted one of the big tables to CSV. At least now it recognized > > it as a table. Yet, the table was very badly formated: > > > > https://mchehab.fedorapeople.org/media-kabi-docs-test/rst_tests/packed-rgb.html > > > > This is how this table should look like: > > https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html > > > > Also, as this table has merged cells at the legend. I've no idea how > > to tell sphinx to do that on csv format. > > > > The RST files are on this git tree: > > https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/ > > Yeah, seems it can't do merged cells in csv. Attached patch converts it > back to grid table format and fixes the table definition. > The html output looks usable, but clearly it is no fun to > work with tables in Sphinx. Yes, the output is OK, but, as you said, working with tables in Sphinx is hard, and using asciiart for the kind of tables we have is not nice. > > Sphinx' latex writer can't handle nested tables, though. Yeah, this is a big trouble that need to be solved if you're willing to use Sphinx. Btw, it crashes when trying to generate man pages: Exception occurred: File "/usr/lib/python2.7/site-packages/docutils/writers/manpage.py", line 627, in depart_entry self._active_table.append_cell(self.body[start:]) AttributeError: 'NoneType' object has no attribute 'append_cell' The full traceback has been saved in /tmp/sphinx-err-04qRMz.log, if you want to report the issue to the developers. So, if we're willing to use sphinx, someone should either fix it to produce latex nexted table and fix it to generate manpages, or we'll need to stick with just html output. > Python's docutils rst2latex can, but that doesn't help here. > rst2pdf also supports it. At least here, rst2* scripts were unable to identify that the index.rst had links to other rst documents. In the specific case of rst2latex, I got several errors like: index.rst:21: (ERROR/3) Unknown interpreted text role "ref". > But I have doubts such a large > table would render OK in pdf without using landscape orientation. Yeah, in the past, when we had pdf enabled for DocBook (e. g. when media development was using a separate mercurial tree), I guess we had tags changing the text orientation on a few tables that would otherwise won't diplay fine, but I can't remember the dirty details anymore. > I have not tried because I used python3-sphinx but rst2pdf > is only availble for Python2 in Debian so it does not integrate > with Sphinx. > > > Johannes -- Thanks, Mauro -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
On Wed, 4 May 2016 13:50:35 -0300 Mauro Carvalho Chehabwrote: > Em Wed, 4 May 2016 19:13:21 +0300 > Jani Nikula escreveu: > > I think we should go for vanilla sphinx at first, to make the setup step > > as easy as possible for everyone. > > Vanilla Sphinx doesn't work, as reST markup language is too limited > to support the docs we have. So, we should either push the needed > extensions to Sphinx reST or find a way to put it at Kernel tree > without causing too much pain for the developers, e. g. as something > that just doing "make htmldoc" would automatically use such extensions > without needing to actually install the extensions. It works for everything except the extended media book, right? Or is there something I'm missing here? I am very much interested in having one system for *all* our docs, but we wouldn't attempt to convert a document can't be expressed in sphinx. > > Even if it means still doing that ugly > > docproc step to call kernel-doc. We can improve from there, and I > > definitely appreciate your work on making this work with sphinx > > extensions. > > I disagree: We should not cause documentation regressions by > producing incomplete documentation and losing tables because of > such conversion. > The quality of the documentation after the change should be *at least* > equal to the one we current produce, never worse. Agreed; that's why I think the existing DocBook mechanism should stay in place until that document will be improved by the change. But I'd rather not hold up the entire process for one book, especially since pushing that process forward can only help in dealing with those final problems. > > That said, how would it work to include the kernel-doc extension in the > > kernel source tree? Having things just work if sphinx is installed is > > preferred over requiring installation of something extra from pypi. (I > > know this may sound backwards for a lot of projects, but for kernel I'm > > pretty sure this is how it should be done.) > > Yeah, using pypi seems an additional undesired step. Aren't there > any way to make python to use an additional extension at the > Kernel tree without needing to install it? Well, sphinx has to come from somewhere too. But yes, we should be able to carry extensions in the kernel tree. But I would still rather upstream it (to sphinx) if possible, so we're not stuck trying to maintain it across several sphinx releases. I don't know how volatile their interfaces are, but it would be, in any case, the analog of an out-of-tree module... jon -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
On Wed, May 4, 2016 at 5:02 PM, Daniel Vetterwrote: > On Wed, May 04, 2016 at 08:57:13AM -0600, Jonathan Corbet wrote: >> On Wed, 4 May 2016 16:18:27 +0200 >> Daniel Vetter wrote: >> >> > > I'd really like to converge on the markup question, so that we can start >> > > using all the cool stuff with impunity in gpu documentations. >> > >> > Aside: If we decide this now I could send in a pull request for the >> > rst/sphinx kernel-doc support still for 4.7 (based upon the minimal >> > markdown/asciidoc code I still have). That would be really awesome ... >> >> Sorry for my relative absence...I'm still busy dealing with bureaucracy >> an ocean away from home. I hope to begin emerging from this mess in the >> near future. >> >> So ... there's the code you have, the work I (+Jani) did, and the work >> Markus has done. Which would you have me push into 4.7? >> >> The sphinx/rst approach does seem, to me, to be the right one, with the >> existing DocBook structure remaining in place for those who want/need >> it. I'm inclined toward my stuff as a base to work with, obviously :) But >> it's hackish at best and needs a lot of cleaning up. It's a proof of >> concept, but it's hardly finished (one might say it's barely begun...) >> >> In the end, I guess, I feel that anything we might try to push for 4.7 is >> going to look rushed and not ready, and Linus might react accordingly. >> I'd be more comfortable aiming for 4.8. I *will* have more time to focus >> on things in that time frame... I suspect you're pretty well fed up with >> this stuff being pushed back, and rightly so. All I can do is apologize. >> >> That said, if you do think there's something out there that is good >> enough to consider pushing in a week or two, do tell and we can all take >> a look. > > Well I'd just have taken the asciidoc hacks I have currently in my > topic/kerneldoc branch, converted to sphinx and looked how it fares. It > should be fairly minimal, and I think the first step we want to do for the > long-term plan. I hope I can ready something, and then we can look whether > it's rushed for 4.7 or not. Ok, discussed this a bit more with Jani on IRC and he really doesn't like the old design of that branch (it calls the converter for every kerneldoc comment). So I guess nothing rushed for 4.7, but hopefully something for 4.8. -Daniel -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
Em Thu, 5 May 2016 07:02:10 -0600 Jonathan Corbetescreveu: > On Wed, 4 May 2016 14:57:38 -0300 > Mauro Carvalho Chehab wrote: > > > Also, media documentation is not just one more documentation. It is > > the biggest one we have, and that has more changes than any other > > documentation under Documentation/DocBook: > > > > $ git lg --since 01/01/2015 ` ls *.tmpl|grep -v media`|wc -l > > 116 > > $ git lg --since 01/01/2015 ` ls *.tmpl|grep media` `find media/ -type > > f`|wc -l > > 179 > > > > It also is more than twice the size of the other DocBook docs: > > > > $ wc -l $(ls *.tmpl|grep media) `find media/ -type f`|tail -1 > > 82275 total > > $ wc -l $(ls *.tmpl|grep -v media)|tail -1 > > 29568 total > > > > E. g. media corresponds to 60% of the number of patches and 73% of > > the DocBook stuff. > > These numbers are not entirely representative, I have to say. You're > ignoring the kerneldoc comments - which is what much of the "DocBook" > documents are made of, and which is the focus of much of this activity. If > you could find a way to count those, you'd get a different picture. Yeah, if we take the big picture, I'm pretty sure that there are more stuff written using kerneldoc. However, what is written at the kerneldoc comments don't use DocBook markup, but its own markup language. Assuming that we'll keep using kerneldoc script to convert from the Kerneldoc NANO markup, it means that any changes at the kerneldoc backend to use DocBook/Markdown/asciidoc/reST/... won't change anything for the developers nor will require them to actually use the new markups. So, they can gradually start using new markups as they wish/learn, if they want to improve the documentation. Lazy developers could just ignore reST if they want, as everything will work as before. However, for the files written directly in DocBook, there's no option: at the moment the doc is converted, the developer will need to submit patches using the new markup language. So, we need to be sure that such transition will happen in a way that won't cause more harm than needed. I would prefer to convert all of them at one, on a single Kernel release (or two), in order to minimize the impact for the developers. > > But I don't think that really matters; there doesn't seem to be *that* much > disagreement here. > > The media book is important; we want it to be a part of the overall kernel > documentation suite and not stuck in some DocBook ghetto. I agree that we > should have an idea for a plausible path for *all* of our documentation. > But I'm also concerned about delaying this work yet again; we have > developers trying to push forward with improved documentation, and they've > had to wait a year for this stuff - so far. Yeah, I understand. Regards, Mauro -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
Jonathan Corbetwrites: > Asciidoc is a credible solution to the formatted documentation problem, > but it's not the only such; I'd like to be sure that we pick the right > one. I worry that asciidoc seems to be aimed mostly at small documents, > and that the project itself seems a little lifeless - it's not a good > sign when your main page's link to the repository has been dead for a long > time. (Asciidoctor seems more active, with the Github folks behind it, > but that means bringing Ruby into the picture). I was surprised when one of the asciidoctor developers said that asciidoc itself was 'in maintenance mode for existing users'. I've tried asciidoctor but never got it to the point where I was happy with the results. Having two tools using the same nominal format doesn't seem like a great idea to me. It's also clear from my hacking in asciidoc that docbook is the expected target for that tool. I've managed to make direct HTML output usable, but LaTeX doesn't work at all. Something which focuses on direct HTML (and ePub) output would be pretty nice. > An alternative we haven't really looked at yet is ReStructuredText (or > "RST") and the Sphinx system (sphinx-doc.org) built on top of it. RST is > YA simple markup scheme, remarkably similar to Markdown or Asciidoc; > Sphinx is a fairly sophisticated documentation system that uses RST. I've installed debian's python3-sphinx package; it looks like it doesn't have a huge dependency chain below it, which is a nice change. I translated a fairly long document from asciidoc to rst using pandoc by using the docbook output from asciidoc -- pandoc doesn't have a native asciidoc reader, only a writer. The result didn't totally suck, although I haven't messed with fixing the css or using a different theme at all. http://keithp.com/~keithp/altusmetrum-sphinx/altusmetrum.html I installed the sphinxcontrib.fulltoc extension so that the whole TOC was visible from each section; this made navigating a lot easier. Having search included (if you have javascript) seems like a nice feature. > Like asciidoc, Sphinx is Python-based, so it adds little to the toolchain > requirements there. Having functional native latex output means that even PDF generation is lighterweight though. > It produces integrated, multi-file HTML natively, > with a TOC, an index, cross-file cross references, and more. It can make > things like function indexes. It claims output in epub, docbook, and man > (I've not yet messed with those). The path to PDF is via latex; clearly > the docbook path could be used too. I've tried epub and latex backends; epub seems just fine (it's just html, after all). LaTeX works, and generates functional PDF, but I'm going to have to spend a bunch of time making it looks nice. http://keithp.com/~keithp/altusmetrum-sphinx/AltusMetrum.pdf > So can we discuss? I'm not saying we have to use Sphinx, but, should we > choose not to, we should do so with open eyes and good reasons for the > course we do take. What do you all think? Having spent the afternoon playing with it, I'm definitely impressed. I've spent a ton of time getting asciidoc to generate html and pdf that I can tolerate; far too much of that involved hacking XML files related to the docbook backend. Pros * Credible HTML output without docbook * Credible PDF output without docbook. * Constructs a unified set of documents across multiple files. * Written in Python (2 or 3) * PanDoc already supports rst for both input and output. So, if we get bored with RST, we've got a way out. Cons * Table formatting doesn't seem as sophisticated as asciidoc Questions * Conditional text appears to be harder to manage (I haven't managed to make it work at all). * Takes over a directory making building more than one document in a directory hard/impossible? The config file must be named 'conf.py'? -- -keith signature.asc Description: PGP signature
Re: Kernel docs: muddying the waters a bit
Em Tue, 08 Mar 2016 11:49:35 +0200 Jani Nikulaescreveu: > On Tue, 08 Mar 2016, Dan Allen wrote: > > One of the key goals of the Asciidoctor project is to be able to directly > > produce a wide variety of outputs from the same source (without DocBook). > > We've added flexibility and best practices into the syntax and matured the > > converter mechanism to bridge this (sometimes very wide) gap. > > I think our conclusion so far was that the native AsciiDoc (and > Asciidoctor) outputs fell short of our needs, forcing us to use the > DocBook pipeline. I, for one, was hoping we could eventually simplify > the toolchain. For example, there was no support for chunked, or split > to chapters, HTML, and the single page result was simply way too big. > > > Asciidoctor is the future of AsciiDoc. Even the AsciiDoc Python maintainers > > acknowledge that (including the original creator). > > Thanks for the input. We've touched the topic of AsciiDoc > vs. Asciidoctor before [1]. So we should be using Asciidoctor instead of > AsciiDoc. That actually makes choosing asciidoc harder, because > requiring another language environment complicates, not simplifies, the > toolchain. I'd really like to lower the bar for building the > documentation, for everyone, so much so that it becomes part of the > normal checks for patch inclusion. As I failed to find a way to solve the issues with Sphinx/RST, I started a PoC using Asciidoctor for the Media uAPI docbook conversion, in order to see if this would work for us. I'm not a big fan of using a language that I don't domain, like Ruby (and the same applies to Python), but, as I said before, provided that the toolchain works, can easily be installed on Fedora and Debian, and provide the functionality I need, I'm ok with that. On my tests, Asciidoctor is really fast. It takes a fraction of the time require to build from DocBook, with is a good thing. Yet, I suspect that it doesn't have the strict checks that xmllint have. For example, I didn't see any warning about a missing cross-ref. We use those warnings to discover if something is added at the code, but were not documented. Its error reports are also crappy, as it doesn't tell where the problem is. For example: $ asciidoctor -n -b docbook media_api.adoc asciidoctor: WARNING: tables must have at least one body row On a document with 33793 lines (this is the size of the uAPI doc when converted to AsciiDoc format), the above warning doesn't help. So, I'm actually planning to use AsciiDoc/xmllint to check for documentation troubles, if possible. Even with AsciiDoc/Asciidoctor, there are several features at the media documents that aren't well supported. For example, we widely use the Docbook's tags to generate a manpage-like description: V4L2 open() v4l2-open Open a V4L2 device #include fcntl.h int open const char *device_name int flags Asciidoctor doesn't produce anything like that, if the booktype is not "manpage". Well, the media documentation is a multi-part book. So, all those tags should be manually converted (actually, pandoc made a mess with those tags - so manual work is required anyway, no matter what markdown language we use). It will be a hard work, but, at least, this is doable. Also, it doesn't accept images with PDF format. We have a few PDF images, although I we have them also in GIF. So, not a big issue here. I also did lots of table conversions, to see if our complex tables would fit. The answer is: Yes: on all cases I converted, it worked fine. There are two ways of doing that: - Nested tables - Merging cells Some tables we use won't work fine with nested tables, as they have cells merging the entire line of such tables. Also, converting some tables to nested tables would be incredible hard, as we have really big tables there that would need to add dozens of nested tables inside, like this one: https://linuxtv.org/downloads/v4l-dvb-apis/extended-controls.html#id-1.4.4.14.6.4 Using merging cells works fine, tough, and it is not hard to use. The tables are easy to edit. The Asciidoctor syntax (not sure if AsciiDoc also accepts this) is: .MFC 5.1 Control IDs [width="100%",cols="7%,40%,13%,40%",options="header",] |=== |ID |Type 2+| Description ... The "2+|" tells that the next content should be merged into two cells. One of the good things is that we're not forced to use asciiart, with would make it really hard to handle the tables. For those wanting to see the results so far: The html is at: https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/media_api.html The testing tree is at: https://git.linuxtv.org/mchehab/asciidoc-poc.git/ PS.: pandoc did a really crap job on the conversion. To convert this
Re: Kernel docs: muddying the waters a bit
On Fri, 04 Mar 2016, Mauro Carvalho Chehabwrote: > Em Thu, 03 Mar 2016 15:23:23 -0800 > Keith Packard escreveu: > >> Mauro Carvalho Chehab writes: >> >> > On my tests, Sphinix seemed too limited to format tables. Asciidoc >> > produced an output that worked better. >> >> Yes, asciidoc has much more flexibility in table formatting, including >> the ability to control text layout within cells and full control over >> borders. >> >> However, I think asciidoc has two serious problems: >> >> 1) the python version (asciidoc) appears to have been abandoned in >> favor of the ruby version. >> >> 2) It really is just a docbook pre-processor. Native html/latex output >> is poorly supported at best, and exposes only a small subset of the >> full capabilities of the input language. >> >> As such, we would have to commit to using the ruby version and either >> committing to fixing the native html output backend or continuing to use >> the rest of the docbook toolchain. >> >> We could insist on using the python version, of course. I spent a bit of >> time hacking that up to add 'real' support for a table-of-contents in >> the native HTML backend and it looks like getting those changes >> upstreamed would be reasonably straightforward. However, we'd end up >> 'owning' the code, and I'm not sure we want to. > > I'm a way more concerned about using a tool that fulfill our needs > than to look for something that won't use the docbook toolchain or > require to install ruby. I think you meant that to be the other way round, or I fail at parsing you. ;) > In the case of Docbook, we know it works and we know already its > issues. Please correct me if I'm wrong, but the big problem we > have is not due to the DocBook toolchain, but due to the lack of > features at the kernel-doc script. Also, xmlto is already installed > by the ones that build the kernel docs. So, keeping use it won't > require to install a weird toolchain by hand. I think kernel-doc is just a small part of the puzzle. It's a problem, but a small one at that, and we've already made it output asciidoc, rst and docbook as part of this exercise. For real, as in code, not as in talk. The reasons I'm involved in this is that I want to make writing documentation and rich kernel-doc comments easier (using lightweight markup) and I want to make building the documentation easier (using a straightforward toolchain with not too many dependencies). I'm hoping the former makes writing documentation more attractive and the latter keeps the documentation and the toolchain in a better shape through having more people actually build the documentation. IMHO docbook is problematic because the toolchain gets too long and fragile. You need plenty of tools installed to build the documentation, it's fussy to get working, and people just won't. Like code, documentation bitrots too when it's not used. The documentation build is broken too often. Debugging formatting issues through the entire pipeline gets hard; I already faced some of this when playing with the kernel-doc->asciidoc->docbook->html chain. In short, I don't think the docbook toolchain fills all of our needs either. > So, to be frank, it doesn't scary me to use either pyhton or > ruby script + docbook. > > Of course, having to own the code has a cost that should be evaluated. > > If, on the other hand, we decide to use RST, we'll very likely need to > patch it to fulfill our needs in order to add proper table support. > I've no idea how easy/difficult would be to do that, nor if Sphinx > upstream would accept such changes. > > So, at the end of the day, we may end by having to carry on our own > version of Sphinx inside our tree, with doesn't sound good, specially > since it is not just a script, but a package with hundreds of > files. If we end up having to modify Sphinx, it has a powerful extension mechanism for this. We wouldn't have to worry about getting it merged to Sphinx upstream, and we wouldn't have to carry a local version of all of Sphinx. (In fact, the extension mechanism provides a future path for doing kernel-doc within Sphinx instead of as a preprocessing step.) I know none of this alleviates your concerns with table supports right now. I'll try to have a look at that a bit more. BR, Jani. -- Jani Nikula, Intel Open Source Technology Center -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
Am 04.05.2016 um 15:43 schrieb Daniel Vetter: > On Wed, May 04, 2016 at 02:40:29PM +0200, Markus Heiser wrote: >>> On Wed, 04 May 2016, Markus Heiser wrote: >>> I'd be *very* hesitant about adding the kind of things you do in >>> reformat_block_rst to kernel-doc. IMO the extraction from kernel-doc >>> comments must be as simple as possible with basically pass-through of >>> the comment blocks to sphinx. >> >> Right, but you forgot, that the current markup in the source code comments >> is based on the kernel-doc-nano-HOWTO and I guess no one will migrate all >> these source code comments to reST markup ;-) >> >> So there is a need to differentiate between *vintage* kernel-doc markup >> and reST markup. >> >> My suggestion is to add a tag to the comments, here a short example >> what this might look like. >> >> -- >> /** >> * drm_minor_release - Release DRM minor >> * @minor: Pointer to DRM minor object >> * >> * Release a minor that was previously acquired via drm_minor_acquire(). >> */ >> -- >> >> in short: the vintage style does not need any change and >> comments with reST markup has a tag ":reST:" in the first >> line(s) ... >> >> -- >> /** >> * :reST: >> * >> * .. c:function:: void drm_minor_release(struct drm_minor *minor) >> * >> *Release DRM minor >> * >> *:param struct drm_minor \*minor: Pointer to DRM minor object >> * >> */ >> -- >> >> Comments with the ":reST:" tag could be exported and pass-through >> to sphinx. >> >>> Specifically, do not attempt to detect and >>> parse elements like lists in kernel-doc. >> >> If your markup in the comments is plain reST, no need to parse, but there >> are markups in the (vintage) comments which made use of individual >> text-markups, e.g. the markup of lists or ASCII-art. >> >> This individual text-markups has not been converted/rendered in the docbook >> output, but I wanted to convert this individuals to reST, to render them in >> Sphinx. > > I think we need to unconfuse what's current standardize kerneldoc markup. > There's three bits: > - The header with the one-liner and parameters, i.e. OK, forget my example, I don't really want to promote this way ... I agree, it is better to stay with standardize kernel-doc markup and provide a "pass through" for the section-content (even if it is mixing markups). Am 04.05.2016 um 15:41 schrieb Jani Nikula : > Kernel-doc the tool should continue to parse kernel-doc comments at the > high level like they currently are, according to the kernel-doc > howto. The requirement was unclear for me, thanks to Daniel and Jani for clarifying this. Am 04.05.2016 um 15:43 schrieb Daniel Vetter : > This is already used widely in kerneldoc included by gpu.tmpl, and > currently it's asciidoc. Those lists and asciiarts though are _not_ > standard kerneldoc at all. But imo any doc toolchain improvement should > integrate along those lines. > > For reference, this is what it's supposed to look like with the asciidoc > support enabled: > > https://dri.freedesktop.org/docs/drm/API-struct-drm-display-mode.html > > Fyi those hacked-up patches to make this happen are available in > > https://cgit.freedesktop.org/drm-intel/log/?h=topic/kerneldoc > Correct my, if I'am wrong. I'am a bit unfamiliar with DOCPROC in particular with your "MARKDOWNREADY := gpu.xml" process. As I understood, you convert markdown to docbook within the kernel-doc script using pandoc's executable? ... I don't face this topic. With my modification of kernel-doc I produced pure reST markup from standardize kernel-doc markup, no intermediate steps or tools required. Am 04.05.2016 um 15:41 schrieb Jani Nikula : > Overall I think we should promote fixing those in the kernel-doc > comments. Trying to guesswork in kernel-doc tool will leave the source > littered with bad examples that get proliferated all around. Instead of > gradually fixing things, we'll end up gradually messing things up even > more. I agree with you, lets drop the reformat_block_rst from my kernel-doc script: * https://github.com/return42/sphkerneldoc/blob/master/scripts/kernel-doc#L1736 and we should have a plain "pass through". Am 04.05.2016 um 17:09 schrieb Jonathan Corbet : > I think all of this makes sense. It would be really nice to have the > directives in the native sphinx language like that. I *don't* think we > need to aim for that at the outset; the docproc approach works until we can > properly get rid of it. What would be *really* nice would be to get > support for the kernel-doc directive into the sphinx upstream. No need for kernel-doc directive in sphinx upstream, later it will be an extension which could be installed by a simple command like "pip install kernel-doc-extensions" or similar. I develop these required extension (and more) within my proof of
Re: Kernel docs: muddying the waters a bit
Hy Jani, Am 04.05.2016 um 18:13 schrieb Jani Nikula: >> Am 04.05.2016 um 17:09 schrieb Jonathan Corbet : >> >>> I think all of this makes sense. It would be really nice to have the >>> directives in the native sphinx language like that. I *don't* think we >>> need to aim for that at the outset; the docproc approach works until we can >>> properly get rid of it. What would be *really* nice would be to get >>> support for the kernel-doc directive into the sphinx upstream. >> >> No need for kernel-doc directive in sphinx upstream, later it will be >> an extension which could be installed by a simple command like >> "pip install kernel-doc-extensions" or similar. >> >> I develop these required extension (and more) within my proof of concept >> on github ... this takes time ... if I finished all my tests and all is >> well, I will build the *kernel-doc-extensions* package and deploy it >> on https://pypi.python.org/pypi from where everyone could install this >> with "pip". > > I think we should go for vanilla sphinx at first, to make the setup step > as easy as possible for everyone. Even if it means still doing that ugly > docproc step to call kernel-doc. We can improve from there, and I > definitely appreciate your work on making this work with sphinx > extensions. +1 > That said, how would it work to include the kernel-doc extension in the > kernel source tree? Having things just work if sphinx is installed is > preferred over requiring installation of something extra from pypi. (I > know this may sound backwards for a lot of projects, but for kernel I'm > pretty sure this is how it should be done.) Thats all right. Lets talk about the extension infrastructure by example: First we have to chose a folder where we place all the *sphinx-documentation* I recommending: /share/linux/Documentation/sphinx Next we have to chose a folder where reST-extensions should take place, I would prefer ... or similar: /share/linux/Documentation/sphinx/extensions Lets say, you wan't to get in use of the "flat-table" extension. Copy (only) the rstFlatTable.py file from my POC extension folder (ignore other extensions which might be there) ... https://github.com/return42/sphkerneldoc/tree/master/doc/extensions Now lets say you are writing on a gpu book, it wold be placed in the folder: /share/linux/Documentation/sphinx/gpu In this gpu-folder you have to place the conf.py config file, needed to setup the sphinx build environment. /share/linux/Documentation/sphinx/gpu/conf.py In this conf.py you have to *register* your folder with the extensions. import os.path, sys EXT_PATH = "../extensions" # the path of extension folder relative to the conf.py file sys.path.insert(0, os.path.join(os.path.dirname(__file__), EXT_PATH))) # now import the "flat-table" extension, it will be self-registering to docutils import rstFlatTable Thats all, you can run your sphinx-build command and the flat-tables in your reST sources should be handled as common tables. ASIDE: You will find similar parts in your conf.py which you have created with the sphinx-quickstart command. There, you will also find a block looks like ... extensions = [ 'sphinx.ext.autodoc' ] Don't try to add flat-table extension to this list. This list is a list of sphinx extensions, we will use it later for other *real* sphinx extensions. A few words about the flat-table extension and a (future) kernel-doc one: The flat-table is a pure docutils (the layer below sphinx) extension which is not application specific, so I will ask for moving it to the docutils upstream. The kernel-doc extension on the other side is a very (very) kernel specific application, this would never go to sphinx nor docutils upstream. --Markus-- -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
Hi Jani, I forget to mentioning, with a local copy of my kernel-doc script: https://github.com/return42/sphkerneldoc/blob/master/scripts/kernel-doc You could do reST markup in the source code comments and extract them. This might be a interim workaround which helps you not to edit source code comments twice. -- Markus-- Am 06.05.2016 um 13:23 schrieb Markus Heiser: > > Hy Jani, > > Am 04.05.2016 um 18:13 schrieb Jani Nikula : > >>> Am 04.05.2016 um 17:09 schrieb Jonathan Corbet : >>> I think all of this makes sense. It would be really nice to have the directives in the native sphinx language like that. I *don't* think we need to aim for that at the outset; the docproc approach works until we can properly get rid of it. What would be *really* nice would be to get support for the kernel-doc directive into the sphinx upstream. >>> >>> No need for kernel-doc directive in sphinx upstream, later it will be >>> an extension which could be installed by a simple command like >>> "pip install kernel-doc-extensions" or similar. >>> >>> I develop these required extension (and more) within my proof of concept >>> on github ... this takes time ... if I finished all my tests and all is >>> well, I will build the *kernel-doc-extensions* package and deploy it >>> on https://pypi.python.org/pypi from where everyone could install this >>> with "pip". >> >> I think we should go for vanilla sphinx at first, to make the setup step >> as easy as possible for everyone. Even if it means still doing that ugly >> docproc step to call kernel-doc. We can improve from there, and I >> definitely appreciate your work on making this work with sphinx >> extensions. > > +1 > >> That said, how would it work to include the kernel-doc extension in the >> kernel source tree? Having things just work if sphinx is installed is >> preferred over requiring installation of something extra from pypi. (I >> know this may sound backwards for a lot of projects, but for kernel I'm >> pretty sure this is how it should be done.) > > Thats all right. Lets talk about the extension infrastructure by example: > > First we have to chose a folder where we place all the *sphinx-documentation* > I recommending: > > /share/linux/Documentation/sphinx > > Next we have to chose a folder where reST-extensions should take place, I > would prefer ... or similar: > > /share/linux/Documentation/sphinx/extensions > > Lets say, you wan't to get in use of the "flat-table" extension. > > Copy (only) the rstFlatTable.py file from my POC extension folder (ignore > other extensions which might be there) ... > > https://github.com/return42/sphkerneldoc/tree/master/doc/extensions > > Now lets say you are writing on a gpu book, it wold be placed in the folder: > > /share/linux/Documentation/sphinx/gpu > > In this gpu-folder you have to place the conf.py config file, needed to > setup the sphinx build environment. > > /share/linux/Documentation/sphinx/gpu/conf.py > > In this conf.py you have to *register* your folder with the extensions. > > > >import os.path, sys > >EXT_PATH = "../extensions" # the path of extension folder relative to > the conf.py file >sys.path.insert(0, os.path.join(os.path.dirname(__file__), EXT_PATH))) > ># now import the "flat-table" extension, it will be self-registering to > docutils > >import rstFlatTable > > > > Thats all, you can run your sphinx-build command and the flat-tables in your > reST sources should be handled as common tables. > > ASIDE: > > You will find similar parts in your conf.py which you have created > with the sphinx-quickstart command. There, you will also find a block > looks like ... > > extensions = [ >'sphinx.ext.autodoc' > > ] > > Don't try to add flat-table extension to this list. This list is a list > of sphinx extensions, we will use it later for other *real* sphinx > extensions. > > A few words about the flat-table extension and a (future) kernel-doc one: > > The flat-table is a pure docutils (the layer below sphinx) extension which > is not application specific, so I will ask for moving it to the docutils > upstream. > > The kernel-doc extension on the other side is a very (very) kernel specific > application, this would never go to sphinx nor docutils upstream. > > --Markus-- > -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
Hi Jonahtan, Am 12.04.2016 um 17:46 schrieb Jonathan Corbet: > On Fri, 8 Apr 2016 17:12:27 +0200 > Markus Heiser wrote: > >> motivated by this MT, I implemented a toolchain to migrate the kernel’s >> DocBook XML documentation to reST markup. >> >> It converts 99% of the docs well ... to gain an impression how >> kernel-docs could benefit from, visit my sphkerneldoc project page >> on github: >> >> http://return42.github.io/sphkerneldoc/ > > So I've obviously been pretty quiet on this recently. Apologies...I've > been dealing with an extended death-in-the-family experience, and there is > still a fair amount of cleanup to be done. > > Looking quickly at this work, it seems similar to the results I got. But > there's a lot of code there that came from somewhere? >From me? ... except the kernel-doc script which is a fork from your git://git.lwn.net/linux.git doc/sphinx > I'd put together a > fairly simple conversion using pandoc and a couple of short sed scripts; > is there a reason for a more complex solution? It depends. If you have a simple DocBook with less various markup, maybe not. May you want to read my remarks about migration tools and especially pandoc: * https://return42.github.io/sphkerneldoc/articles/dbtools.html#remarks-on-pandoc A few more words about, what I have done: I wrote a lib of XML filters which might be also usefully in other migration projects (dbxml). * https://github.com/return42/sphkerneldoc/blob/master/scripts/dbxml.py It uses a xml-parser, pandoc, pandoc-filters and regular expressions. Because I did not implemented a whole converter, I hacked around pandoc. Thats why conversion is done in several steps: 1. copy xml file(s) to a cache space 2. substitude unsolved internal and external entities 3. filter all xml files * run custom hooks on every node * apply filters on every node and inject reST into the XML-tree where pandoc fails. https://github.com/return42/sphkerneldoc/blob/master/scripts/dbxml.py#L515 4. convert intermediary XML result with pandoc to json (needed by pandoc filters) 5. apply pandoc-filter and clean up the injected reST markup from step3 6. convert filtered json to reST 7. fix the produce reST with regular expression ... the last step is similar to your sed scripts. And I wrote a commandline Interface to use this lib (see func db2rst): * https://github.com/return42/sphkerneldoc/blob/master/scripts/dbtools.py#L146 With this db2rst all kernel DB-XML books could be migrated, except the linux-tv book, which has much more complexity. For this, there is a separated commandline called media2rst * https://github.com/return42/sphkerneldoc/blob/master/scripts/dbtools.py#L107 The media2rst needs several special handlings, which is implemented in hooks (the dbxml interface method) * https://github.com/return42/sphkerneldoc/blob/master/scripts/media.py Summarize, why should one prefer this tools over pandoc + sed? * Pandoc coverage is less on reading and writing, this is where dbxml comes into play - reading DocBook: https://github.com/jgm/pandoc/blob/master/src/Text/Pandoc/Readers/DocBook.hs#L23 - writing reST has many bugs and leaks (you fixed some of them with sed) * Pandoc does not support external entities (linux-tv), covered by dbxml * dbxml brings the ability to chunk one large XML book into small reST chunks e.g. kernel-hacking book: https://github.com/return42/sphkerneldoc/tree/master/doc/books/kernel-hacking * dbxml lets you manipulate the XML source before you convert it to reST this might helpfull e.g. if you have to convert single-column informal-tables to lists or other things ... in short; dbxml and it's hooks are the key to hack everything you need in a full automated DocBook-->reST migration workflow. --Markus-- > Thanks for looking into this, anyway; I hope to be able to focus more on > it shortly. > > jon > -- > To unsubscribe from this list: send the line "unsubscribe linux-media" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel docs: muddying the waters a bit
Em Fri, 6 May 2016 18:26:10 +0200 Markus Heiserescreveu: > Hi Mauro, > > Am 06.05.2016 um 13:03 schrieb Mauro Carvalho Chehab > : > > Yeah, it looks better, however table truncation seem to be > > happening also on other parts, like the tables on this page: > > > > > > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-packed-rgb.html > > (original table: > > https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html) > > This table should contain 32 bits, but only the first 7 bits are shown > > > > and those (among others): > > > > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-y41p.html > > > > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/dev-sliced-vbi.html > > > > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html > > > > Hmm... after looking more carefully, it added a horizontal scroll bar. > > That looks ugly, IMHO, and makes harder to understand its contents. The > > last one, in particular > > (https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html), > > is a big table on both horiz and vert dimensions. Try to read how the > > bits are packed on a random line in the middle of the table, like > > MEDIA_BUS_FMT_BGR565_2X8_LE and you'll understand what I mean. > > I know what you mean ;-) ... I'am also unhappy, but I will address this point > later when it goes to finish the layout. > > Currently lets focus on contend and (the two)extensions. OK. > > The table here looks weird (although it is correct): > > > > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-srggb10p.html > > (original table: > > https://linuxtv.org/downloads/v4l-dvb-apis/pixfmt-srggb10p.html) > > > > I validated this and the other tables you mentioned above ... these are > all correct migrated ... it is 1:1 translated from DocBook ... they > might be show different to this what you know from your docbook > toolchain, because in the docbook-html you have no table grids and > wrong cellspans are not clear ... sometimes, like in the last example > you gave: > > > > > > a colspec might ambiguous ... so there is no clear role to migrate. Ok, that's what I was thinking. Ok, this can be fixed later manually, where needed. Of course one way would be to disable grids on those tables, but I would instead fix it. > > > It seems that Sphinx is assuming something like "A4 portrait" for > > the margins, while those big tables would only fit (in PDF) as > > "A4 landscape". > > No, no, no ;-) > > Sphinx assumes nothing about the layout, sphinx and the underlying > docutils mostly juggling with nodes and the writers in e.g. the > html-writer, outputs a clear HTML without any style but with classified > HTML tags. Styling is done in the presentation layer, in HTML, it is > done in CSS. Hmm... Then there's something deadly wrong at CSS template, as it is shown texts only at half of my horizontal res (1920). Probably this is the culpit: .container { margin: 50px auto 40px auto; width: 600px; text-align: center; } width is set to 600px, instead of using a percentage, like 100% (or 90%). > > > I guess the better would be to not limit the right > > margin or to change it where those big tables happen, in order to > > allow PDF generation. > > Generating PDF has nothing to do with generating HTML. To generate > PDF there is a other writer, the latex2e writer, which produce > LaTeX markup from which you build PDF or other printed-like medias. Ok. > > > > > There are also some tables that went wrong. See the Color Sample > > Location table at: > > > > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-yuv444m.html > > I'm pretty sure we'll need to fix some cases like that manually. > > I didn't check, but perhaps, in this case, the DocBook were using > > empty columns just to make the table bigger. If so, this is not a > > problem with the conversion, and should be manually fixed later. > > +1 > > Yes, lets do it manually later ... what I have in my POC is a automated > process, it is hard to consider individuals in an automatic process. > Making details *nicer* and making ambiguous markups clear is manually > done in minutes where I need hours to implement this in a automated > process. Yeah, we should not try to fix everything via auto-scripts, and spending time right now with manual fixes will be wasted, as we need to run it at the latest media documentation, as changes might have happened upstream. > > Aside, from: http://docutils.sourceforge.net/docs/peps/pep-0258.html > > Docutils Project Model -- Project components and data flow: > > +---+ > |Docutils: | > | docutils.core.Publisher, | >
Re: Kernel docs: muddying the waters a bit
Hi all, hi Jonathan, Am 06.05.2016 um 15:42 schrieb Mauro Carvalho Chehab: > Em Fri, 6 May 2016 15:32:35 +0200 > Markus Heiser escreveu: > >> Hi Mauro, >> >> Am 06.05.2016 um 13:35 schrieb Mauro Carvalho Chehab >> : >> >>> Markus, >>> >>> Em Fri, 6 May 2016 13:23:06 +0200 >>> Markus Heiser escreveu: >>> In this conf.py you have to *register* your folder with the extensions. A few words about the flat-table extension and a (future) kernel-doc one: >>> >>> ... >>> The flat-table is a pure docutils (the layer below sphinx) extension which is not application specific, so I will ask for moving it to the docutils upstream. >>> >>> So, if I understood well, your proposal is to have a conf.py and the >>> flat-table (plus other extensions) at the Kernel tree. >> >> Each book (better call it root-node) is a Sphinx-project, each >> Sphinx-project need a conf.py file (the build configuration file) >> in its reST-source tree. >> >> * http://www.sphinx-doc.org/en/stable/config.html >> >>> Assuming that docutils upstream receives the flat-table extension >>> (and eventually modifies it), while the new version doesn't arrive >>> all distros, we'll end by having some developers using the newer >>> docutils with the extension, plus others using the in-tree one. >>> >>> Is there a way to specify at the conf.py what extension variant >>> should it use, in case of both the in-tree or the docutils have >>> the same? >> >> The build configuration file is a regular python file, you can >> implement conditions whatever you want/need. >> >>> This could be trickier if they end by modifying the extension, >>> but we can always backport the latest version, if they change the >>> API. >> >> As far as i know, the docutils API is stable since 2002. In the >> meantime there has been so many application build on it that >> it is not realistic, you will see a not backward compatibly >> change. >> >> The docutils project is conservative, very conservative, IMO to >> conservative. >> >> Today I'am doubtful if it isn't better I would merge it sphinx >> upstream. I have to discuss this with some maintainers, but >> before I have to persuade myself, that all aspects are covered >> and the implementations are robust. We are at the beginning and >> we should not fear about every bit which could happen in the future. >> >> The sphinx / docutils bottom plate gives us a number of degrees >> of freedom to find answers to question we have not yet asked. ;-) > > Ok. So, from what I understand, once Sphinx support is added at > Kernel upstream, we could convert the media docbook to > reST+flat-table extension, adding such extension either on a shared > place or only for the media DocBook build, together with its > conf.py. > Yes, in your media-conf you could decide to use a extension. > Once it reaches upstream (either sphinx or docutils), we can > work to make it integrate better with upstream as needed. > > Right? yes, right :-) > If so, I'm ok with merging it as soon as possible. If we advice a merge of the flat-table directive we should bundle this with the (to implement) "kernel-doc" directive ... > In reST the directive might look like: > > - > Device Instance and Driver Handling > === > > .. kernel-doc:: drivers/gpu/drm/drm_drv.c > :doc: driver instance overview > :exported: > > - and the patches from my kernel-doc perl script to produce reST from source code comments. With this bundle within the kernel tree we have a good starting point to compose reST documents from scratch and to migrate book by book from DocBook to reST. I insist to migrate book by book, because there are some broken books. Broken by that, that some sources have changed but not the corresponding documentation which use the comments of these sources ... grap "Ooops" in the builded (xml or rst) docs. E.g. I greped the .rst file and found the following Oops in the migrated books: ./books/mtdnand/pubfunctions-000-012.rst:13:Oops ./books/scsi/mid_layer-000-001-016-003.rst:13:Oops ./books/s390-drivers/ccw-000-004-003.rst:13:Oops ./books/device-drivers/devdrivers-000-003-048.rst:13:Oops ./books/device-drivers/devdrivers-000-003-050.rst:13:Oops ./books/device-drivers/Basics-000-001-002.rst:13:Oops ./books/device-drivers/devdrivers-000-003-031.rst:13:Oops ./books/device-drivers/Basics-000-009-032.rst:13:Oops ./books/kernel-api/kernel-lib-000-004-008.rst:13:Oops ./books/gadget/api-000-011-005.rst:13:Oops ./books/gadget/api-000-011-009.rst:13:Oops ./books/gadget/api-000-011-007.rst:13:Oops ./books/gadget/api-000-011-003.rst:13:Oops ./books/gadget/api-000-011-011.rst:13:Oops ./books/genericirq/intfunctions-000-009.rst:13:Oops - Summarize @Jonathan: what do you think? Should I prepare a patch with a basic reST (sphinx) build
Re: Kernel docs: muddying the waters a bit
Em Fri, 6 May 2016 16:27:21 +0200 Markus Heiserescreveu: > Hi all, hi Jonathan, > > Am 06.05.2016 um 15:42 schrieb Mauro Carvalho Chehab > : > > > Em Fri, 6 May 2016 15:32:35 +0200 > > Markus Heiser escreveu: > > > >> Hi Mauro, > >> > >> Am 06.05.2016 um 13:35 schrieb Mauro Carvalho Chehab > >> : > >> > >>> Markus, > >>> > >>> Em Fri, 6 May 2016 13:23:06 +0200 > >>> Markus Heiser escreveu: > >>> > > In this conf.py you have to *register* your folder with the extensions. > A few words about the flat-table extension and a (future) kernel-doc > one: > >>> > >>> ... > >>> > The flat-table is a pure docutils (the layer below sphinx) extension > which > is not application specific, so I will ask for moving it to the docutils > upstream. > >>> > >>> So, if I understood well, your proposal is to have a conf.py and the > >>> flat-table (plus other extensions) at the Kernel tree. > >> > >> Each book (better call it root-node) is a Sphinx-project, each > >> Sphinx-project need a conf.py file (the build configuration file) > >> in its reST-source tree. > >> > >> * http://www.sphinx-doc.org/en/stable/config.html > >> > >>> Assuming that docutils upstream receives the flat-table extension > >>> (and eventually modifies it), while the new version doesn't arrive > >>> all distros, we'll end by having some developers using the newer > >>> docutils with the extension, plus others using the in-tree one. > >>> > >>> Is there a way to specify at the conf.py what extension variant > >>> should it use, in case of both the in-tree or the docutils have > >>> the same? > >> > >> The build configuration file is a regular python file, you can > >> implement conditions whatever you want/need. > >> > >>> This could be trickier if they end by modifying the extension, > >>> but we can always backport the latest version, if they change the > >>> API. > >> > >> As far as i know, the docutils API is stable since 2002. In the > >> meantime there has been so many application build on it that > >> it is not realistic, you will see a not backward compatibly > >> change. > >> > >> The docutils project is conservative, very conservative, IMO to > >> conservative. > >> > >> Today I'am doubtful if it isn't better I would merge it sphinx > >> upstream. I have to discuss this with some maintainers, but > >> before I have to persuade myself, that all aspects are covered > >> and the implementations are robust. We are at the beginning and > >> we should not fear about every bit which could happen in the future. > >> > >> The sphinx / docutils bottom plate gives us a number of degrees > >> of freedom to find answers to question we have not yet asked. ;-) > > > > Ok. So, from what I understand, once Sphinx support is added at > > Kernel upstream, we could convert the media docbook to > > reST+flat-table extension, adding such extension either on a shared > > place or only for the media DocBook build, together with its > > conf.py. > > > > Yes, in your media-conf you could decide to use a extension. > > > Once it reaches upstream (either sphinx or docutils), we can > > work to make it integrate better with upstream as needed. > > > > Right? > > yes, right :-) > > > If so, I'm ok with merging it as soon as possible. > > If we advice a merge of the flat-table directive we should > bundle this with the (to implement) "kernel-doc" directive ... > > > In reST the directive might look like: > > > > - > > Device Instance and Driver Handling > > === > > > > .. kernel-doc:: drivers/gpu/drm/drm_drv.c > > :doc: driver instance overview > > :exported: > > > > - > > and the patches from my kernel-doc perl script to produce > reST from source code comments. > > With this bundle within the kernel tree we have a good starting > point to compose reST documents from scratch and to migrate book > by book from DocBook to reST. > > I insist to migrate book by book, because there are some > broken books. Broken by that, that some sources have changed > but not the corresponding documentation which use the comments > of these sources ... grap "Ooops" in the builded (xml or rst) docs. > > E.g. I greped the .rst file and found the following Oops in the migrated > books: > > ./books/mtdnand/pubfunctions-000-012.rst:13:Oops > ./books/scsi/mid_layer-000-001-016-003.rst:13:Oops > ./books/s390-drivers/ccw-000-004-003.rst:13:Oops > ./books/device-drivers/devdrivers-000-003-048.rst:13:Oops > ./books/device-drivers/devdrivers-000-003-050.rst:13:Oops > ./books/device-drivers/Basics-000-001-002.rst:13:Oops > ./books/device-drivers/devdrivers-000-003-031.rst:13:Oops > ./books/device-drivers/Basics-000-009-032.rst:13:Oops >
Re: Kernel docs: muddying the waters a bit
Hi kernel-doc authors, motivated by this MT, I implemented a toolchain to migrate the kernel’s DocBook XML documentation to reST markup. It converts 99% of the docs well ... to gain an impression how kernel-docs could benefit from, visit my sphkerneldoc project page on github: http://return42.github.io/sphkerneldoc/ The sources available at: https://github.com/return42/sphkerneldoc The work is underway, suggestions are welcome! .. have a nice weekend .. --M-- Am 13.03.2016 um 16:33 schrieb Markus Heiser: > > Am 10.03.2016 um 16:21 schrieb Mauro Carvalho Chehab > : > >> Em Thu, 10 Mar 2016 12:25:58 +0200 >> Jani Nikula escreveu: >> >>> TL;DR? Skip to the last paragraph. >>> >>> On Wed, 09 Mar 2016, Mauro Carvalho Chehab wrote: I guess the conversion to asciidoc format is now in good shape, at least to demonstrate that it is possible to use this format for the media docbook. Still, there are lots of broken references. >>> >>> Getting references right with asciidoc is a big problem in the >>> kernel-doc side. As I wrote before, the proofs of concept only worked >>> because everything was processed as one big file (via includes). The >>> Asciidoctor inter-document references won't help, because we won't know >>> the target document name while processing kernel-doc. >> >> I was able to produce chunked htmls here with: >> >> asciidoctor -b docbook45 media_api.adoc >> xmlto -o html-dir html media_api.xml >> >> The results are at: >> >> https://mchehab.fedorapeople.org/media-kabi-docs-test/asciidoc_tests/chunked/ >> >> But yeah, all references seem to be broken there. It could be due to some >> conversion issue (I didn't actually tried to check what's wrong there), >> but I think that there's something not ok with docbook45 >> output for multi-part documents (on both AsciiDoc and Asciidoctor). >> >>> Sphinx is massively better at handling cross references for >>> kernel-doc. We can use domains (C language) and roles (e.g. functions, >>> types, etc.) for the references, which provide kind of >>> namespaces. Sphinx warns for referencing non-existing targets, but >>> doesn't generate broken links in the result like Asciidoctor does. >>> >>> For example, in the documentation for a function that has struct foo as >>> parameter or return type, a cross reference to struct foo is added >>> automagically, but only if documentation for struct foo actually >>> exists. In Asciidoctor, we would have to blindly generate the references >>> ourselves, and try to resolve broken links ourselves by somehow >>> post-processing the result. >>> Yet, from my side, if we're willing to get rid of DocBook, then Asciidoctor seems to be the *only* alternative so far to parse the complex media documents. >>> >>> I think you mean, "get rid of DocBook as source format", not altogether? >>> I'm yet to be convinved we could rely on Asciidoctor's native formats. >> >> What I mean is that, right now, I see only two alternatives for the >> media uAPI documentation: >> 1) keep using DocBook; >> 2) AsciiDoc/Asciidoctor. >> >> Sphinx doesn't have what's needed to support the complexity of the >> media books, specially since cell span seems to be possible only >> by using asciiArt formats. Writing a big table using asciiArt is >> something that is a *real pain*. Also, as tested, if the table is >> too big, it fails to parse such asciiArt tables. So, while Sphinx >> doesn't have a decent way to describe tables, we can't use it. > > > Huge tables and cell-spans are the *real pain* ;-) ... with sphinx-doc, > (mostly) you have more then one choice .. e.g. import csv tables .. > but this should be discussed by example ... > > >> If it starts implementing it, then we can check if the other >> features used by the media documentation are also supported. >> Probably, multi-part books would be another pain with Sphinx. >> We have actually 4 books inside a common body. A few chapters >> (like book licensing, bibliography, error codes) are shared >> by all 4 documents. >> >> But, so far, I can't see any way to port media books without >> lots of lot of work to develop new features at the Sphinx code. > > > may I can help you ... > > >>> The toolchain gets faster, easier to debug and simplified a lot with >>> DocBook out of the equation completely. Sphinx itself is stable, widely >>> available, and well documented. IMO there's sufficient native output >>> format support. There are plenty of really nice extensions >>> available. There's a possibility of doing kernel-doc as an extension in >>> the future (either by calling current kernel-doc from the extension or >>> by rewriting it). >> >> Well, if we go to Sphinx for kernel-doc, that means that we'll need >> 2 different tools for the documentation: >> - Sphinx for kernel-doc >> - either DocBook or
Re: Kernel docs: muddying the waters a bit
Hi Mauro, hi kernel-doc authors, Am 12.04.2016 um 15:58 schrieb Mauro Carvalho Chehab: > Em Fri, 8 Apr 2016 17:12:27 +0200 > Markus Heiser escreveu: > >> Hi kernel-doc authors, >> >> motivated by this MT, I implemented a toolchain to migrate the kernel’s >> DocBook XML documentation to reST markup. >> >> It converts 99% of the docs well ... to gain an impression how >> kernel-docs could benefit from, visit my sphkerneldoc project page >> on github: >> >> http://return42.github.io/sphkerneldoc/ >> >> The sources available at: >> >> https://github.com/return42/sphkerneldoc >> >> The work is underway, suggestions are welcome! >> >> .. have a nice weekend .. > > Hi Markus, > > Thanks for your work. I'm basically worried about the media docbook, > with has some complexities that I was not able to figure out a way to > convert it to reST. > > So, let me pinpoint some issues that I noticed there, on a quick > look. > > The first thing I noticed is that the index doesn't match what's > there, when generated from DocBook: > https://linuxtv.org/downloads/v4l-dvb-apis/v4l2spec.html > > So, for instance, "Interfaces" is at the same level as "Input/Output". > > This sounds like an something went wrong when dealing with the title > indentation levels during the conversion. > Yes, the hierarchical structure was broken by two causes. First was, that the *chunking* was wrong and the other was, that my DocBook-XML-filter (dbxml) placed all sections and sub(-sub)-sections at the same level (direct under chapter). This was not only broken in the linux_tv book, in the other books also. Thanks for pointing and sorry that I have overlooked this (I was so much focused on on converting single elements to reST, that I not see the wood for the trees). It is now fixed, may you give it a new try. > I would also like to see the titles numbered in chapters (and, if > possible, in sections and items) and to be properly indented at the > index. BTW a few words about differences between DockBook and reST (Sphinx). With DocBook you write *books*, the protocol (the DocBook application) has no facility to *chunk* and interconnect several documents. The external ENTITY is a workaround on the SGML layer, not on XML nor on the DB-application layer. Thats the reason, why so many XML-tools don't handle this entities and many DocBook to (e.g.) reST tools are fail. With **standard** reST it is nearly the same, except there is a "include" directive on the application layer. But this directive is very simple, comparable to the C preprocessor "#include" directive. With the **superset** reST-markup of Sphinx-doc you get a the "toctree" directive, which lets you control how a document-tree should be build. http://www.sphinx-doc.org/en/stable/markup/toctree.html @Mauro: you mentioned a docutils (rst2*) experience in your mail http://marc.info/?l=linux-doc=145735316012094=2 Because the "toctree" directive -- and other directives we use -- are a part of a superset of the **standard** reST, the standard docutils (like rst2*) will not work. OK, back to your requirements: within the toctree directive you can set options like "maxdepth" and "numbered". It is a decision, how deep TOCs should go and if they should be numbered. IMO, in a HTML rendered page, with a proper navigation bar on the side, deep TOCs in the running text have no pros, they only blow up the running text and bring more scrolling with. In my sense numbering chapters make only sense in books, not in HTML pages, where you have hyperlinks. Just for demonstration, I added numbering in the linux-tv book: https://github.com/return42/sphkerneldoc/commit/468ded71f62d497ac71aead1a6d50de7ef77c3c3 May be, I will drop it later, because all reST sources are generated by a make target and I always commit the whole reST tree. As I said, it is a decision which might be made later, when the migration takes places. > Also, it seems that there's still a lot of work to do, as there are several > tables that are missing conversion, like the table for "struct > v4l2_pix_format": > > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt.html > > and the big tables at: > > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-packed-rgb.html > > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-packed-yuv.html > > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/subdev-formats.html > Yes, I marked them as TODO: https://github.com/return42/sphkerneldoc/blob/master/scripts/media.py#L262 aspect *authoring tables* (see below) > Also, some tables that are not so big like: > > https://return42.github.io/sphkerneldoc/books/linux_tv/media/v4l/pixfmt-y41p.html > > don't look nice, at least on my browser, as the "white" area is too small, > and some cells seem to be broken in
Re: Kernel docs: muddying the waters a bit
On Sat, Mar 05, 2016 at 11:29:37PM -0300, Mauro Carvalho Chehab wrote: > > I converted one of the big tables to CSV. At least now it recognized > it as a table. Yet, the table was very badly formated: > > https://mchehab.fedorapeople.org/media-kabi-docs-test/rst_tests/packed-rgb.html > > This is how this table should look like: > https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html > > Also, as this table has merged cells at the legend. I've no idea how > to tell sphinx to do that on csv format. > > The RST files are on this git tree: > https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/ Yeah, seems it can't do merged cells in csv. Attached patch converts it back to grid table format and fixes the table definition. The html output looks usable, but clearly it is no fun to work with tables in Sphinx. Sphinx' latex writer can't handle nested tables, though. Python's docutils rst2latex can, but that doesn't help here. rst2pdf also supports it. But I have doubts such a large table would render OK in pdf without using landscape orientation. I have not tried because I used python3-sphinx but rst2pdf is only availble for Python2 in Debian so it does not integrate with Sphinx. Johannes >From 61674b398e778bd5ff644ffd493d5ff1cfaca0ef Mon Sep 17 00:00:00 2001 From: Johannes StezenbachDate: Sun, 6 Mar 2016 23:55:19 +0100 Subject: [PATCH] some progress for html output --- _static/borderless.css | 3 -- _static/v4l2tables.css | 9 + _templates/layout.html | 9 + packed-rgb.rst | 88 +- pixfmt-yuyv.rst| 2 +- v4l-table-within-table.rst | 72 +++-- 6 files changed, 105 insertions(+), 78 deletions(-) delete mode 100644 _static/borderless.css create mode 100644 _static/v4l2tables.css diff --git a/_static/borderless.css b/_static/borderless.css deleted file mode 100644 index bfd4b01..000 --- a/_static/borderless.css +++ /dev/null @@ -1,3 +0,0 @@ -table#table-borderless { -border: 1px solid black; -} diff --git a/_static/v4l2tables.css b/_static/v4l2tables.css new file mode 100644 index 000..c045e45 --- /dev/null +++ b/_static/v4l2tables.css @@ -0,0 +1,9 @@ +table.noborder { +border: 1px solid black; +background: white; +white-space: nowrap; +} + +table.noborder td, table.noborder th { +padding: 0px; +} diff --git a/_templates/layout.html b/_templates/layout.html index b6bf12b..637332d 100644 --- a/_templates/layout.html +++ b/_templates/layout.html @@ -1,9 +1,2 @@ {% extends "!layout.html" %} -{% block tables %} - -table#table-borderless { -border: 1px solid black; -} - -{{ super() }} -{% endblock %} +{% set css_files = css_files + ["_static/v4l2tables.css"] %} diff --git a/packed-rgb.rst b/packed-rgb.rst index 352b91c..b4fcf3e 100644 --- a/packed-rgb.rst +++ b/packed-rgb.rst @@ -9,25 +9,46 @@ graphics frame buffers. They occupy 8, 16, 24 or 32 bits per pixel. These are all packed-pixel formats, meaning all the data for a pixel lie next to each other in memory. -.. csv-table:: Table: Packed RGB Image Formats - :header: Identifier,Code, ,Byte 0 in memory,Byte 1,Byte 2,Byte 3 +.. table:: Packed RGB Image Formats + :class: noborder - ``V4L2_PIX_FMT_RGB332``,'RGB1',,r\ :sub:`2`,r\ :sub:`1`,r\ :sub:`0`,g\ :sub:`2`,g\ :sub:`1`,g\ :sub:`0`,b\ :sub:`1`,b\ :sub:`0` - ``V4L2_PIX_FMT_ARGB444``,'AR12',,g\ :sub:`3`,g\ :sub:`2`,g\ :sub:`1`,g\ :sub:`0`,b\ :sub:`3`,b\ :sub:`2`,b\ :sub:`1`,b\ :sub:`0`,,a\ :sub:`3`,a\ :sub:`2`,a\ :sub:`1`,a\ :sub:`0`,r\ :sub:`3`,r\ :sub:`2`,r\ :sub:`1`,r\ :sub:`0` - ``V4L2_PIX_FMT_XRGB444``,'XR12',,g\ :sub:`3`,g\ :sub:`2`,g\ :sub:`1`,g\ :sub:`0`,b\ :sub:`3`,b\ :sub:`2`,b\ :sub:`1`,b\ :sub:`0`,,-,-,-,-,r\ :sub:`3`,r\ :sub:`2`,r\ :sub:`1`,r\ :sub:`0` - ``V4L2_PIX_FMT_ARGB555``,'AR15',,g\ :sub:`2`,g\ :sub:`1`,g\ :sub:`0`,b\ :sub:`4`,b\ :sub:`3`,b\ :sub:`2`,b\ :sub:`1`,b\ :sub:`0`,,a,r\ :sub:`4`,r\ :sub:`3`,r\ :sub:`2`,r\ :sub:`1`,r\ :sub:`0`,g\ :sub:`4`,g\ :sub:`3` - ``V4L2_PIX_FMT_XRGB555``,'XR15',,g\ :sub:`2`,g\ :sub:`1`,g\ :sub:`0`,b\ :sub:`4`,b\ :sub:`3`,b\ :sub:`2`,b\ :sub:`1`,b\ :sub:`0`,,-,r\ :sub:`4`,r\ :sub:`3`,r\ :sub:`2`,r\ :sub:`1`,r\ :sub:`0`,g\ :sub:`4`,g\ :sub:`3` - ``V4L2_PIX_FMT_RGB565``,'RGBP',,g\ :sub:`2`,g\ :sub:`1`,g\ :sub:`0`,b\ :sub:`4`,b\ :sub:`3`,b\ :sub:`2`,b\ :sub:`1`,b\ :sub:`0`,,r\ :sub:`4`,r\ :sub:`3`,r\ :sub:`2`,r\ :sub:`1`,r\ :sub:`0`,g\ :sub:`5`,g\ :sub:`4`,g\ :sub:`3` - ``V4L2_PIX_FMT_ARGB555X``,'AR15' | (1<<31),,a,r\ :sub:`4`,r\ :sub:`3`,r\ :sub:`2`,r\ :sub:`1`,r\ :sub:`0`,g\ :sub:`4`,g\ :sub:`3`,,g\ :sub:`2`,g\ :sub:`1`,g\ :sub:`0`,b\ :sub:`4`,b\ :sub:`3`,b\ :sub:`2`,b\ :sub:`1`,b\ :sub:`0` - ``V4L2_PIX_FMT_XRGB555X``,'XR15' | (1<<31),,-,r\ :sub:`4`,r\ :sub:`3`,r\ :sub:`2`,r\ :sub:`1`,r\ :sub:`0`,g\ :sub:`4`,g\ :sub:`3`,,g\ :sub:`2`,g\ :sub:`1`,g\ :sub:`0`,b\ :sub:`4`,b\ :sub:`3`,b\ :sub:`2`,b\ :sub:`1`,b\ :sub:`0` -
Re: Kernel docs: muddying the waters a bit
Em Mon, 18 Apr 2016 10:10:17 +0200 Markus Heiserescreveu: > Hi Mauro, hi kernel-doc authors, > > Am 12.04.2016 um 15:58 schrieb Mauro Carvalho Chehab > : > > > Em Fri, 8 Apr 2016 17:12:27 +0200 > > Markus Heiser escreveu: > > > >> Hi kernel-doc authors, > >> > >> motivated by this MT, I implemented a toolchain to migrate the kernel’s > >> DocBook XML documentation to reST markup. > >> > >> It converts 99% of the docs well ... to gain an impression how > >> kernel-docs could benefit from, visit my sphkerneldoc project page > >> on github: > >> > >> http://return42.github.io/sphkerneldoc/ > >> > >> The sources available at: > >> > >> https://github.com/return42/sphkerneldoc > >> > >> The work is underway, suggestions are welcome! > >> > >> .. have a nice weekend .. > > > > Hi Markus, > > > > Thanks for your work. I'm basically worried about the media docbook, > > with has some complexities that I was not able to figure out a way to > > convert it to reST. > > > > So, let me pinpoint some issues that I noticed there, on a quick > > look. > > > > The first thing I noticed is that the index doesn't match what's > > there, when generated from DocBook: > > https://linuxtv.org/downloads/v4l-dvb-apis/v4l2spec.html > > > > So, for instance, "Interfaces" is at the same level as "Input/Output". > > > > This sounds like an something went wrong when dealing with the title > > indentation levels during the conversion. > > > > Yes, the hierarchical structure was broken by two causes. First was, that the > *chunking* was wrong and the other was, that my DocBook-XML-filter (dbxml) > placed > all sections and sub(-sub)-sections at the same level (direct under chapter). > This was > not only broken in the linux_tv book, in the other books also. > > Thanks for pointing and sorry that I have overlooked this (I was so much > focused on > on converting single elements to reST, that I not see the wood for the trees). > > It is now fixed, may you give it a new try. Thanks! It looks good now. > > I would also like to see the titles numbered in chapters (and, if > > possible, in sections and items) and to be properly indented at the > > index. > > BTW a few words about differences between DockBook and reST (Sphinx). > > With DocBook you write *books*, the protocol (the DocBook application) has > no facility to *chunk* and interconnect several documents. The external > ENTITY > is a workaround on the SGML layer, not on XML nor on the DB-application layer. > Thats the reason, why so many XML-tools don't handle this entities and > many DocBook to (e.g.) reST tools are fail. > > With **standard** reST it is nearly the same, except there is a "include" > directive on the application layer. But this directive is very simple, > comparable to the C preprocessor "#include" directive. > > With the **superset** reST-markup of Sphinx-doc you get a the "toctree" > directive, > which lets you control how a document-tree should be build. > > http://www.sphinx-doc.org/en/stable/markup/toctree.html > > @Mauro: you mentioned a docutils (rst2*) experience in your mail > http://marc.info/?l=linux-doc=145735316012094=2 > > Because the "toctree" directive -- and other directives > we use -- are a part of a superset of the **standard** > reST, the standard docutils (like rst2*) will not work. > > OK, back to your requirements: within the toctree directive you can > set options like "maxdepth" and "numbered". It is a decision, how > deep TOCs should go and if they should be numbered. IMO, in a > HTML rendered page, with a proper navigation bar on the side, deep > TOCs in the running text have no pros, they only blow up the running > text and bring more scrolling with. In my sense numbering chapters > make only sense in books, not in HTML pages, where you have hyperlinks. > > Just for demonstration, I added numbering in the linux-tv book: > > https://github.com/return42/sphkerneldoc/commit/468ded71f62d497ac71aead1a6d50de7ef77c3c3 > > May be, I will drop it later, because all reST sources are generated > by a make target and I always commit the whole reST tree. As I said, > it is a decision which might be made later, when the migration takes > places. This is the uAPI spec DocBook, that we modify frequently, as we add more features to the Kernel, and as we make sure that all drivers will behave the same. So, from time to time, we need to clarify some topics at the documentation. By having a numeration, it is easier for us to discuss things like: "1.2.10.14. V4L2_PIX_FMT_VYUY (‘VYUY’)" is not properly described and requires some sort of clarification. Ok, one could also refer to it via a hyperlink, but several Kernel media maintainers prefer to generate a single big html file, as it makes easier to locate everything it is needed on it. So, with the item number, they can just seek