Re: [RFC PATCH 0/5] DT binding documents using text markup
On Tue, Sep 8, 2015 at 9:36 AM, Matt Porter wrote: > On Tue, Sep 01, 2015 at 01:24:17PM -0500, Rob Herring wrote: >> On Tue, Sep 1, 2015 at 1:03 PM, Tim Bird wrote: >> > On Tue, Sep 1, 2015 at 10:35 AM, Frank Rowand >> > wrote: >> >> On 9/1/2015 10:14 AM, Tim Bird wrote: >> >>> On Fri, Aug 28, 2015 at 10:13 AM, Matt Porter >> >>> wrote: >> >> >> >> < snip > >> >> >> >> But to answer your question, if we get a format I'll do >> conversions and hope I'm not alone. >> >>> >> >>> I'm sure others will help out. I will, and I'm pretty sure we can get >> >>> some conversion sprints set up at conferences (I know I can set aside >> >>> some time or resources at ELC in the spring - it might be too late for >> >>> ELCE in October to set up a scheduled block of time, but we can start >> >>> getting the word out.) As I said in my other e-mail, one doesn't have >> >>> to be a kernel coder to do this, and the conversions should be pretty >> >>> straight-forward. >> >>> -- Tim >> >>> >> >> >> >> A conversion sprint at ELCE sounds like a good idea if we can find a >> >> good time to schedule it. I can help, so there will be at least two >> >> of us who can help people as they encounter issues. >> > >> > Even if we don't find a block of time, we can do something like >> > announce a "contest", ask people to do something in their spare time, >> > and find some way to get them a raffle ticket if they submit a patch >> > with a conversion. Then hold an extra prize drawing during the >> > closing session, with just those raffle tickets, and give someone a >> > nice award for contributing. Of course, there's always the adage that >> > you should be careful what you measure and reward... We don't want a >> > flood of crappy conversions, with a time constraint on the review. >> > I'll think some more about this. An alternative would be to have a >> > contest announced ahead of the event, with enough time for people to >> > submit and get reviewed. >> >> Sounds like a review nightmare. That's another reason why as much >> automated conversion we can do, the better. > > I don't want to discount the value of interested people getting together > f2f to review these and potentially clean them up for submission. That > depends on what we thinking is the minimal "in progress" conversion > that can be place upstream. i.e. is it simply compatible strings > autoconverted to tags and the entire current document in comments? That is sufficient for me (I reserve the right to change my mind). Logistically, it needs to be a script that can be run before/during the merge window and perhaps again after. I'd guess the long pole here is how we validate the .yaml files. >> > By the way - I presume the new docs will replace the existing ones, >> > but I could imagine wanting to have them live side-by-side >> > temporarily. Any thoughts on this? Will file name or location >> > changes be allowed during the conversion? >> >> I proposed some ideas earlier in the thread. Either we can have both >> side by side or do a mass conversion to YAML making the existing doc a >> comment (add # prefix). >> > > Were you thinking that this automated conversion would be sufficient > for an initial commit? I'm not sure if I misunderstood in your separate > comments and was looking at this as something that would be hand edited > to move the existing doc (# prefix) into description tags where > appropriate. Yes, I think it is more important to have infrastructure in place to enable others than how much is converted initially. Certainly any changes based on existing docs will have to be manual, but we may be able to do multiple passes of automated conversions. Sharing any conversion scripts is important to enable others for that as well. >> Any renames/moving should be separate (there's some clean-up I'd like >> to there as well). Exact rules depend on the approach, but we need to >> be able to tell which bindings conversions are not started, in >> progress, or complete. > > If we add .yaml in place we can identify what's in progress by the fact > that a .yaml exists with the same filename and then based on which > tags have been populated (such as type: and constraints: not yet > populated) then we know the state. You mean keeping the .txt and .yaml until done? If we are copying in the current doc, then I'd rather not do that. I don't think looking at the tags will work as it would be hard to distinguish incomplete from done. However, we could simply have an "in-progress" tag that is removed when done (might as well take advantage of our new found structured docs). Rob -- To unsubscribe from this list: send the line "unsubscribe devicetree" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH 0/5] DT binding documents using text markup
On Tue, Sep 01, 2015 at 01:24:17PM -0500, Rob Herring wrote: > On Tue, Sep 1, 2015 at 1:03 PM, Tim Bird wrote: > > On Tue, Sep 1, 2015 at 10:35 AM, Frank Rowand > > wrote: > >> On 9/1/2015 10:14 AM, Tim Bird wrote: > >>> On Fri, Aug 28, 2015 at 10:13 AM, Matt Porter > >>> wrote: > >> > >> < snip > > >> > > But to answer your question, if we get a format I'll do > conversions and hope I'm not alone. > >>> > >>> I'm sure others will help out. I will, and I'm pretty sure we can get > >>> some conversion sprints set up at conferences (I know I can set aside > >>> some time or resources at ELC in the spring - it might be too late for > >>> ELCE in October to set up a scheduled block of time, but we can start > >>> getting the word out.) As I said in my other e-mail, one doesn't have > >>> to be a kernel coder to do this, and the conversions should be pretty > >>> straight-forward. > >>> -- Tim > >>> > >> > >> A conversion sprint at ELCE sounds like a good idea if we can find a > >> good time to schedule it. I can help, so there will be at least two > >> of us who can help people as they encounter issues. > > > > Even if we don't find a block of time, we can do something like > > announce a "contest", ask people to do something in their spare time, > > and find some way to get them a raffle ticket if they submit a patch > > with a conversion. Then hold an extra prize drawing during the > > closing session, with just those raffle tickets, and give someone a > > nice award for contributing. Of course, there's always the adage that > > you should be careful what you measure and reward... We don't want a > > flood of crappy conversions, with a time constraint on the review. > > I'll think some more about this. An alternative would be to have a > > contest announced ahead of the event, with enough time for people to > > submit and get reviewed. > > Sounds like a review nightmare. That's another reason why as much > automated conversion we can do, the better. I don't want to discount the value of interested people getting together f2f to review these and potentially clean them up for submission. That depends on what we thinking is the minimal "in progress" conversion that can be place upstream. i.e. is it simply compatible strings autoconverted to tags and the entire current document in comments? > > By the way - I presume the new docs will replace the existing ones, > > but I could imagine wanting to have them live side-by-side > > temporarily. Any thoughts on this? Will file name or location > > changes be allowed during the conversion? > > I proposed some ideas earlier in the thread. Either we can have both > side by side or do a mass conversion to YAML making the existing doc a > comment (add # prefix). > Were you thinking that this automated conversion would be sufficient for an initial commit? I'm not sure if I misunderstood in your separate comments and was looking at this as something that would be hand edited to move the existing doc (# prefix) into description tags where appropriate. > Any renames/moving should be separate (there's some clean-up I'd like > to there as well). Exact rules depend on the approach, but we need to > be able to tell which bindings conversions are not started, in > progress, or complete. If we add .yaml in place we can identify what's in progress by the fact that a .yaml exists with the same filename and then based on which tags have been populated (such as type: and constraints: not yet populated) then we know the state. -Matt -- To unsubscribe from this list: send the line "unsubscribe devicetree" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH 0/5] DT binding documents using text markup
On Tue, Sep 01, 2015 at 11:03:50AM -0700, Tim Bird wrote: > On Tue, Sep 1, 2015 at 10:35 AM, Frank Rowand wrote: > > On 9/1/2015 10:14 AM, Tim Bird wrote: > >> On Fri, Aug 28, 2015 at 10:13 AM, Matt Porter wrote: > > > > < snip > > > > >>> > >>> But to answer your question, if we get a format I'll do > >>> conversions and hope I'm not alone. > >> > >> I'm sure others will help out. I will, and I'm pretty sure we can get > >> some conversion sprints set up at conferences (I know I can set aside > >> some time or resources at ELC in the spring - it might be too late for > >> ELCE in October to set up a scheduled block of time, but we can start > >> getting the word out.) As I said in my other e-mail, one doesn't have > >> to be a kernel coder to do this, and the conversions should be pretty > >> straight-forward. > >> -- Tim > >> > > > > A conversion sprint at ELCE sounds like a good idea if we can find a > > good time to schedule it. I can help, so there will be at least two > > of us who can help people as they encounter issues. > > Even if we don't find a block of time, we can do something like > announce a "contest", ask people to do something in their spare time, > and find some way to get them a raffle ticket if they submit a patch > with a conversion. Then hold an extra prize drawing during the > closing session, with just those raffle tickets, and give someone a > nice award for contributing. Of course, there's always the adage that > you should be careful what you measure and reward... We don't want a > flood of crappy conversions, with a time constraint on the review. > I'll think some more about this. An alternative would be to have a > contest announced ahead of the event, with enough time for people to > submit and get reviewed. > > By the way - I presume the new docs will replace the existing ones, > but I could imagine wanting to have them live side-by-side > temporarily. Any thoughts on this? Will file name or location > changes be allowed during the conversion? Good question. As a starting point I simply changed the file extension and they live in the same directory structure with the .txt versions. I can't think of an advantage to moving locations or file names initially. Certainly some file names could use improvement but changing them introduces confusion. -Matt -- To unsubscribe from this list: send the line "unsubscribe devicetree" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH 0/5] DT binding documents using text markup
On Tue, Sep 01, 2015 at 10:35:30AM -0700, Frank Rowand wrote: > On 9/1/2015 10:14 AM, Tim Bird wrote: > > On Fri, Aug 28, 2015 at 10:13 AM, Matt Porter wrote: > > < snip > > > >> > >> But to answer your question, if we get a format I'll do > >> conversions and hope I'm not alone. > > > > I'm sure others will help out. I will, and I'm pretty sure we can get > > some conversion sprints set up at conferences (I know I can set aside > > some time or resources at ELC in the spring - it might be too late for > > ELCE in October to set up a scheduled block of time, but we can start > > getting the word out.) As I said in my other e-mail, one doesn't have > > to be a kernel coder to do this, and the conversions should be pretty > > straight-forward. > > -- Tim > > > > A conversion sprint at ELCE sounds like a good idea if we can find a > good time to schedule it. I can help, so there will be at least two > of us who can help people as they encounter issues. I'll be there as well. -Matt -- To unsubscribe from this list: send the line "unsubscribe devicetree" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH 0/5] DT binding documents using text markup
On Tue, Sep 01, 2015 at 10:14:06AM -0700, Tim Bird wrote: > On Fri, Aug 28, 2015 at 10:13 AM, Matt Porter wrote: > > > Now, I would caution about trying to do too much on Day 1 or we > > could end up back at the "never doing anything" stage. > > I strongly agree with that (despite my having just sent a bunch of comments > on the binding format document). > > > It would > > be an improvement to simply check that the basic tags exist as > > shown in the [R] or [O] fields in the documentation. One thing > > I should point out is that I carefully avoided marking some tags > > as [R] where existing bindings don't have them...even if logically, > > a description should be required on every binding. The idea here > > is to avoid updating content at the same time that we are updating > > the format. > > I strongly agree with this as well. I think that the goal for the first > pass should be "no lost text". That is, if it doesn't convert into something > in the binding format, it should still be preserved somewhere, for evaluation > and formatting on a subsequent pass. > > > Rather, I think it would be better to get the base > > format updated, then come back with a janitorial team and add > > descriptions (since now we can generate a worklist of those > > bindings missing a top-level description) and systematically > > fix those and review with the appropriate maintainers. > > Agreed. It will be very useful, once we get our first pass done, to > be able to find: 1) documents not converted yet, 2) property definitions > that are missing certain fields, 3) things that could still be formatted, > from text that is remaining in the description strings or other free-form > text. > This should be easy to script. Exactly, if we were to avoid populating type: and constraints: flags then it is very easy to identify those bindings not yet converted to using those tags. > >> An example such as checking that compatible strings are documented as > >> checkpatch.pl does would be nice. Roughly, that would be just list all > >> compatible values. > > > > Ok, so my comments above were strictly about a validator for the > > binding doc submission itself. I can add an example based on your > > checkpatch.pl to adapt it to the .yaml compatible tags. > > > >> > The scope of the initial YAML DT Binding format was specifically > >> > limited to supporting *only* the content we have in bindings today. > >> > The idea here is to propose and agree on something that will take > >> > us just a few steps in the right direction. If we move *all* current > >> > binding content to a machine parseable format, additional features > >> > can be added with more automation and scripting. As it stands today, > >> > because of the inconsistency of the wording of the files, we can't > >> > add a lot of new features to the content until we convert what we > >> > have today into a standard format. > >> > > >> > With that said, it should be noted that some new features such as > >> > "type" tags to indicate cell types could be added to support > >> > additional DTS validation beyond what the current content supports. > >> > Another possibility is adding "range" type information to validate > >> > the legal values for a cell. > >> > > >> > This series is broken up into three major parts: > >> > > >> > 1) The documentation defining the YAML DT binding format > >> > 2) A skeleton device binding example illustrating use of this format > >> > 3) Some real binding conversions (eeprom.txt, phy-bindings.txt, and > >> >ti-phy.txt > >> > > >> > As a proof of concept of what can be done with a proper machine > >> > readable DT binding source file, there's a simple markdown document > >> > generator at https://github.com/konsulko/dtgendoc. Also, to see > >> > actual output from the generator, the generated markdown from those > >> > bindings is viewable at https://github.com/konsulko/dtgendoc/wiki > >> > >> Nice. > >> > >> > There's a lot of other possibilities for validation tools using > >> > only the data we have today in the bindings. In addition, Frank > >> > Rowand covered some DT debug techniques that would benefit from > >> > the binding documentation being 100% reliably searchable. > >> > > >> > I found it useful to see a side-by-side view of a converted doc > >> > versus the original content, so here's a screenshot of eeprom.txt > >> > vs. eeprom.yaml: > >> > https://github.com/konsulko/dtgendoc/wiki#eepromtxt-vs-eepromyaml > >> > > >> > When we decide on a text markup format that is acceptable, then the > >> > next step is to convert all the bindings. That process would start > >> > with the complete set of generic bindings as they will be referenced > >> > by the actual device bindings. > >> > >> You are going to do that for everyone, right? ;) > > > > Let's just say that I'm banking on others helping here once we have > > a format agreed upon. If we can hold the binding doc schema definition > > initially to just define tags for content t
Re: [RFC PATCH 0/5] DT binding documents using text markup
On Mon, Aug 31, 2015 at 09:05:00AM -0500, Rob Herring wrote: > On Fri, Aug 28, 2015 at 12:13 PM, Matt Porter wrote: > > On Fri, Aug 28, 2015 at 09:26:17AM -0500, Rob Herring wrote: > >> On Fri, Aug 28, 2015 at 12:23 AM, Matt Porter wrote: > >> > During the Device Tree microconference at Linux Plumbers 2015, we had > >> > a short discussion about how to improve DT Binding Documentation. A > > [...] > > >> > One caveat with YAML is it does not tolerate tabs. Yes, I said it. > >> > No tabs! This can be managed with proper editor modes and also with > >> > helper scripts to strip tabs to aid in people passing planned > >> > checkpatch.pl checks that would run YAML DT Binding specific tag > >> > validators for new bindings. > >> > >> What do parsers do with tabs? Throw an error? > > > > Yes, they throw an error. Keep in mind that most of what I used to start > > are general purpose conversion tools on top of a particular scripting > > languages's high level binding to libyaml. The error output leaves a > > bit to be desired for our use case. In any case, when I was developing > > the skeleton.yaml I used the yaml script from > > https://github.com/ryo/yamltools to catch all these syntax errors I > > was inserting..like tabs. the PyYaml binding being used in my PoC > > dtgendoc does the same thing but I don't gracefully handle those > > errors like we could. > > > >> Beyond tabs, how do we check files can be parsed both generically and > >> for any binding specific requirements. We now need a schema for > >> checking the schema. We need some equivalent to compile testing. > > > > Right. So, I think what you are touching on is something I should > > have expanded on in the TODO list. Basically, we need a scripted > > tool that we run from checkpatch.pl that 1) reads the .yaml and > > validates the YAML itself (that comes for free in the high level > > parsers) reporting errors in a sensical manner 2) validates our > > DT binding specific tags > > We all know that no one runs checkpatch.pl. ;) I really want the basic > checking of the doc files to run from make (and run by 0-day). The > tool dependency could be an issue though. However, DocBook builds from > make and I don't think many people check that regularly. Then there is > using the binding docs to check dts files. That should probably be > part of the dtb building. Ok, makes sense. We can do that. I get this uneasy feeling about what happens to all of this *when* bindings and dts files move out of the kernel. > > Now, I would caution about trying to do too much on Day 1 or we > > could end up back at the "never doing anything" stage. > > Certainly, but I would like to have a plan for what Day 2 and 3 look like. Sure. > > > It would > > be an improvement to simply check that the basic tags exist as > > shown in the [R] or [O] fields in the documentation. One thing > > I should point out is that I carefully avoided marking some tags > > as [R] where existing bindings don't have them...even if logically, > > a description should be required on every binding. The idea here > > is to avoid updating content at the same time that we are updating > > the format. Rather, I think it would be better to get the base > > format updated, then come back with a janitorial team and add > > descriptions (since now we can generate a worklist of those > > bindings missing a top-level description) and systematically > > fix those and review with the appropriate maintainers. > > Yes. Any checking would be a great improvement. > > [...] > > >> > When we decide on a text markup format that is acceptable, then the > >> > next step is to convert all the bindings. That process would start > >> > with the complete set of generic bindings as they will be referenced > >> > by the actual device bindings. > >> > >> You are going to do that for everyone, right? ;) > > > > Let's just say that I'm banking on others helping here once we have > > a format agreed upon. If we can hold the binding doc schema definition > > initially to just define tags for content that already exists in our > > textual binding docs, the effort for conversion is tolerable. To give > > an example, that phy-bindings.txt, it took 15 minutes to convert and > > and pass through the yaml parser and dtgendoc. The reason is that it's > > pure reformatting work. It doesn't take any special knowledge of the > > hardware and it doesn't involve reviewing dts files to extra > > additional information. Some of the annoyances can be streamlined > > like tab stripping and handling the two space indentation to make > > this process faster. One of my next things is to get a simple tool > > going that reports problems with conversions, essentially what I > > said was needed to integrate with checkpatch, so this process of > > conversion is even faster. Trivial peripheral bindings like eeprom.txt > > can be done in 5 minutes or so right now. > > What if instead of using the docs as a starting point, we use dts > files
Re: [RFC PATCH 0/5] DT binding documents using text markup
On Tue, Sep 01, 2015 at 09:14:24AM -0700, Tim Bird wrote: > On Thu, Aug 27, 2015 at 10:23 PM, Matt Porter wrote: > > During the Device Tree microconference at Linux Plumbers 2015, we had > > a short discussion about how to improve DT Binding Documentation. A > > number of issues were raised (again, as these things have been > > discussed in the past) including: > > > > * Inconsistency between binding documents due to prose text > > format. > > * Inability to reliably machine read bindings for mass update > > or search. > > * Bit rot of bindings as new conventions are agreed upon but > > only new bindings are changed. > > > > Grant Likely probably summed up the issue best with "...as long as > > bindings are human readable, we'll have issues...". > I'd rephrase that to "... as long as bindings are not machine readable, > we'll have issues...". Human readable and machine readable are not > mutually exclusive. XML is not, IMHO, human readable, but YAML looks > pretty good, so I think it's a good choice. Yes, that's exactly what I hoped to find with this experiment...that we can have both. > > The context > > of that comment was, of course, regarding our current documents > > written in very inconsistent prose style. When the topic of needing > > the bindings in a rigid format was raised, there was general head > > nodding that this was needed. It was noted that this has been > > discussed many times before and nothing has been done. > > > > My proposed solution to the problem is to convert all DT bindings > > a rigid text markup format. In choosing a text markup language my > > requirements were: > > > > 1) Human readable > > 2) Well documented > > 3) Easy to translate to other data formats > > 4) Well supported by tools and libraries > > > > After looking at a number of markup options, YAML stood out as the > > one that meets all of these requirements. The YAML syntax is adopted > > in many projects specifically because of the high level of readability. > > A comprehensive spec is at http://www.yaml.org/spec/1.2/spec.html. > > There's a number of tools to convert between YAML and other popular > > data formats such as JSON and XML. XML was cited by Behan Webster > > during the microconference as an important data format as the type > > of developers that may produce comprehensive DTS Binding validation > > tools will want to use XML. Every major scripting language has a > > high level binding to the low level libyaml C library to facilitate > > handling of YAML data files. > > > > One caveat with YAML is it does not tolerate tabs. Yes, I said it. > > No tabs! This can be managed with proper editor modes and also with > > helper scripts to strip tabs to aid in people passing planned > > checkpatch.pl checks that would run YAML DT Binding specific tag > > validators for new bindings. > > This will be an endless source of pain. Based on your examples, > it looks like we only need a very small subset of YAML. I'd be willing > to write a parser for the language, but we might lose the ability to > do conversions to other formats. It depends on how comprehensive > and automatic the helper scripts to strip tabs would be. We need something to validate that a binding meets our binding schema so it shouldn't a huge problem to have it warn and by option, reformat the binding with tabs stripped. Having an appropriate mode available for vim/emacs would go a long way to helping here too. > > The scope of the initial YAML DT Binding format was specifically > > limited to supporting *only* the content we have in bindings today. > > The idea here is to propose and agree on something that will take > > us just a few steps in the right direction. If we move *all* current > > binding content to a machine parseable format, additional features > > can be added with more automation and scripting. As it stands today, > > because of the inconsistency of the wording of the files, we can't > > add a lot of new features to the content until we convert what we > > have today into a standard format. > > > > With that said, it should be noted that some new features such as > > "type" tags to indicate cell types could be added to support > > additional DTS validation beyond what the current content supports. > > Another possibility is adding "range" type information to validate > > the legal values for a cell. > > I would recommend we add type and something for value constraints. > Even if we don't add populate those immediately, I think thinking > through some of the possible uses will be helpful to avoid doing > something dumb. > 'choices' or 'enums' seem frequently used. I see lots of "must be one > of..." in the binding docs I read. Yes, so definitely after initial feedback, we're going to define some kind of type and constraint tags. Whether we use C-like expressions for constraints or something else is still up for debate. And as you p
Re: [RFC PATCH 0/5] DT binding documents using text markup
On Tue, Sep 1, 2015 at 1:03 PM, Tim Bird wrote: > On Tue, Sep 1, 2015 at 10:35 AM, Frank Rowand wrote: >> On 9/1/2015 10:14 AM, Tim Bird wrote: >>> On Fri, Aug 28, 2015 at 10:13 AM, Matt Porter wrote: >> >> < snip > >> But to answer your question, if we get a format I'll do conversions and hope I'm not alone. >>> >>> I'm sure others will help out. I will, and I'm pretty sure we can get >>> some conversion sprints set up at conferences (I know I can set aside >>> some time or resources at ELC in the spring - it might be too late for >>> ELCE in October to set up a scheduled block of time, but we can start >>> getting the word out.) As I said in my other e-mail, one doesn't have >>> to be a kernel coder to do this, and the conversions should be pretty >>> straight-forward. >>> -- Tim >>> >> >> A conversion sprint at ELCE sounds like a good idea if we can find a >> good time to schedule it. I can help, so there will be at least two >> of us who can help people as they encounter issues. > > Even if we don't find a block of time, we can do something like > announce a "contest", ask people to do something in their spare time, > and find some way to get them a raffle ticket if they submit a patch > with a conversion. Then hold an extra prize drawing during the > closing session, with just those raffle tickets, and give someone a > nice award for contributing. Of course, there's always the adage that > you should be careful what you measure and reward... We don't want a > flood of crappy conversions, with a time constraint on the review. > I'll think some more about this. An alternative would be to have a > contest announced ahead of the event, with enough time for people to > submit and get reviewed. Sounds like a review nightmare. That's another reason why as much automated conversion we can do, the better. > By the way - I presume the new docs will replace the existing ones, > but I could imagine wanting to have them live side-by-side > temporarily. Any thoughts on this? Will file name or location > changes be allowed during the conversion? I proposed some ideas earlier in the thread. Either we can have both side by side or do a mass conversion to YAML making the existing doc a comment (add # prefix). Any renames/moving should be separate (there's some clean-up I'd like to there as well). Exact rules depend on the approach, but we need to be able to tell which bindings conversions are not started, in progress, or complete. Rob -- To unsubscribe from this list: send the line "unsubscribe devicetree" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH 0/5] DT binding documents using text markup
On Tue, Sep 1, 2015 at 10:35 AM, Frank Rowand wrote: > On 9/1/2015 10:14 AM, Tim Bird wrote: >> On Fri, Aug 28, 2015 at 10:13 AM, Matt Porter wrote: > > < snip > > >>> >>> But to answer your question, if we get a format I'll do >>> conversions and hope I'm not alone. >> >> I'm sure others will help out. I will, and I'm pretty sure we can get >> some conversion sprints set up at conferences (I know I can set aside >> some time or resources at ELC in the spring - it might be too late for >> ELCE in October to set up a scheduled block of time, but we can start >> getting the word out.) As I said in my other e-mail, one doesn't have >> to be a kernel coder to do this, and the conversions should be pretty >> straight-forward. >> -- Tim >> > > A conversion sprint at ELCE sounds like a good idea if we can find a > good time to schedule it. I can help, so there will be at least two > of us who can help people as they encounter issues. Even if we don't find a block of time, we can do something like announce a "contest", ask people to do something in their spare time, and find some way to get them a raffle ticket if they submit a patch with a conversion. Then hold an extra prize drawing during the closing session, with just those raffle tickets, and give someone a nice award for contributing. Of course, there's always the adage that you should be careful what you measure and reward... We don't want a flood of crappy conversions, with a time constraint on the review. I'll think some more about this. An alternative would be to have a contest announced ahead of the event, with enough time for people to submit and get reviewed. By the way - I presume the new docs will replace the existing ones, but I could imagine wanting to have them live side-by-side temporarily. Any thoughts on this? Will file name or location changes be allowed during the conversion? -- Tim -- To unsubscribe from this list: send the line "unsubscribe devicetree" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH 0/5] DT binding documents using text markup
On 9/1/2015 10:14 AM, Tim Bird wrote: > On Fri, Aug 28, 2015 at 10:13 AM, Matt Porter wrote: < snip > >> >> But to answer your question, if we get a format I'll do >> conversions and hope I'm not alone. > > I'm sure others will help out. I will, and I'm pretty sure we can get > some conversion sprints set up at conferences (I know I can set aside > some time or resources at ELC in the spring - it might be too late for > ELCE in October to set up a scheduled block of time, but we can start > getting the word out.) As I said in my other e-mail, one doesn't have > to be a kernel coder to do this, and the conversions should be pretty > straight-forward. > -- Tim > A conversion sprint at ELCE sounds like a good idea if we can find a good time to schedule it. I can help, so there will be at least two of us who can help people as they encounter issues. -Frank -- To unsubscribe from this list: send the line "unsubscribe devicetree" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH 0/5] DT binding documents using text markup
On Sun, Aug 30, 2015 at 03:04:20PM -0700, Frank Rowand wrote: > On 8/27/2015 10:23 PM, Matt Porter wrote: > > During the Device Tree microconference at Linux Plumbers 2015, we had > > a short discussion about how to improve DT Binding Documentation. A > > number of issues were raised (again, as these things have been > > discussed in the past) including: > > > > * Inconsistency between binding documents due to prose text > > format. > > * Inability to reliably machine read bindings for mass update > > or search. > > * Bit rot of bindings as new conventions are agreed upon but > > only new bindings are changed. > > > > Grant Likely probably summed up the issue best with "...as long as > > bindings are human readable, we'll have issues...". The context > > of that comment was, of course, regarding our current documents > > written in very inconsistent prose style. When the topic of needing > > the bindings in a rigid format was raised, there was general head > > nodding that this was needed. It was noted that this has been > > discussed many times before and nothing has been done. > > > > My proposed solution to the problem is to convert all DT bindings > > a rigid text markup format. In choosing a text markup language my > > requirements were: > > > > 1) Human readable > > 2) Well documented > > 3) Easy to translate to other data formats > > 4) Well supported by tools and libraries > > Thanks for taking this on! The proposal looks like a great start. > The requirements look good. > > > > After looking at a number of markup options, YAML stood out as the > > one that meets all of these requirements. The YAML syntax is adopted > > in many projects specifically because of the high level of readability. > > A comprehensive spec is at http://www.yaml.org/spec/1.2/spec.html. > > There's a number of tools to convert between YAML and other popular > > data formats such as JSON and XML. XML was cited by Behan Webster > > during the microconference as an important data format as the type > > of developers that may produce comprehensive DTS Binding validation > > tools will want to use XML. Every major scripting language has a > > high level binding to the low level libyaml C library to facilitate > > handling of YAML data files. > > > > One caveat with YAML is it does not tolerate tabs. Yes, I said it. > > No tabs! This can be managed with proper editor modes and also with > > helper scripts to strip tabs to aid in people passing planned > > checkpatch.pl checks that would run YAML DT Binding specific tag > > validators for new bindings. > > I'm just a casual user of some markup languages, nowhere near knowledgeable > or an expert, so will be listening to reasoned opinions on the choice of > YAML. But a quick look at the spec you referenced and the examples you > provided gives me some confidence that YAML will work. > > > > The scope of the initial YAML DT Binding format was specifically > > limited to supporting *only* the content we have in bindings today. > > The idea here is to propose and agree on something that will take > > us just a few steps in the right direction. If we move *all* current > > Yes, this is a great approach so we can start making reasonable progress. > (Caveat below.) > > > binding content to a machine parseable format, additional features > > can be added with more automation and scripting. As it stands today, > > because of the inconsistency of the wording of the files, we can't > > add a lot of new features to the content until we convert what we > > have today into a standard format. > > > > With that said, it should be noted that some new features such as > > "type" tags to indicate cell types could be added to support > > additional DTS validation beyond what the current content supports. > > Another possibility is adding "range" type information to validate > > the legal values for a cell. > > The caveat: I think we should look at some obvious features (such as > "type" that you mention) to make sure that YAML and the chosen > structuring will easily and effectively support the features. > > Thinking about how some features might be implemented led to comments > that I will make in replies to the other patches. > > I agree that we should not attempt to add the additional information > enabled by the new features if the information is not in the present > bindings. Adding the new information in a subsequent pass seems to be > a good strategy. Ok, I think Rob had a good point that we need to get the format for these additional pieces of information in place even if we don't populate them to start. Sounds like there's some agreement on this approach. > > > > > This series is broken up into three major parts: > > > > 1) The documentation defining the YAML DT binding format > > 2) A skeleton device binding example illustrating use of this format > > 3) Some real binding conversions (eeprom.txt, phy-bindings.txt,
Re: [RFC PATCH 0/5] DT binding documents using text markup
On Fri, Aug 28, 2015 at 10:13 AM, Matt Porter wrote: > Now, I would caution about trying to do too much on Day 1 or we > could end up back at the "never doing anything" stage. I strongly agree with that (despite my having just sent a bunch of comments on the binding format document). > It would > be an improvement to simply check that the basic tags exist as > shown in the [R] or [O] fields in the documentation. One thing > I should point out is that I carefully avoided marking some tags > as [R] where existing bindings don't have them...even if logically, > a description should be required on every binding. The idea here > is to avoid updating content at the same time that we are updating > the format. I strongly agree with this as well. I think that the goal for the first pass should be "no lost text". That is, if it doesn't convert into something in the binding format, it should still be preserved somewhere, for evaluation and formatting on a subsequent pass. > Rather, I think it would be better to get the base > format updated, then come back with a janitorial team and add > descriptions (since now we can generate a worklist of those > bindings missing a top-level description) and systematically > fix those and review with the appropriate maintainers. Agreed. It will be very useful, once we get our first pass done, to be able to find: 1) documents not converted yet, 2) property definitions that are missing certain fields, 3) things that could still be formatted, from text that is remaining in the description strings or other free-form text. This should be easy to script. >> An example such as checking that compatible strings are documented as >> checkpatch.pl does would be nice. Roughly, that would be just list all >> compatible values. > > Ok, so my comments above were strictly about a validator for the > binding doc submission itself. I can add an example based on your > checkpatch.pl to adapt it to the .yaml compatible tags. > >> > The scope of the initial YAML DT Binding format was specifically >> > limited to supporting *only* the content we have in bindings today. >> > The idea here is to propose and agree on something that will take >> > us just a few steps in the right direction. If we move *all* current >> > binding content to a machine parseable format, additional features >> > can be added with more automation and scripting. As it stands today, >> > because of the inconsistency of the wording of the files, we can't >> > add a lot of new features to the content until we convert what we >> > have today into a standard format. >> > >> > With that said, it should be noted that some new features such as >> > "type" tags to indicate cell types could be added to support >> > additional DTS validation beyond what the current content supports. >> > Another possibility is adding "range" type information to validate >> > the legal values for a cell. >> > >> > This series is broken up into three major parts: >> > >> > 1) The documentation defining the YAML DT binding format >> > 2) A skeleton device binding example illustrating use of this format >> > 3) Some real binding conversions (eeprom.txt, phy-bindings.txt, and >> >ti-phy.txt >> > >> > As a proof of concept of what can be done with a proper machine >> > readable DT binding source file, there's a simple markdown document >> > generator at https://github.com/konsulko/dtgendoc. Also, to see >> > actual output from the generator, the generated markdown from those >> > bindings is viewable at https://github.com/konsulko/dtgendoc/wiki >> >> Nice. >> >> > There's a lot of other possibilities for validation tools using >> > only the data we have today in the bindings. In addition, Frank >> > Rowand covered some DT debug techniques that would benefit from >> > the binding documentation being 100% reliably searchable. >> > >> > I found it useful to see a side-by-side view of a converted doc >> > versus the original content, so here's a screenshot of eeprom.txt >> > vs. eeprom.yaml: >> > https://github.com/konsulko/dtgendoc/wiki#eepromtxt-vs-eepromyaml >> > >> > When we decide on a text markup format that is acceptable, then the >> > next step is to convert all the bindings. That process would start >> > with the complete set of generic bindings as they will be referenced >> > by the actual device bindings. >> >> You are going to do that for everyone, right? ;) > > Let's just say that I'm banking on others helping here once we have > a format agreed upon. If we can hold the binding doc schema definition > initially to just define tags for content that already exists in our > textual binding docs, the effort for conversion is tolerable. To give > an example, that phy-bindings.txt, it took 15 minutes to convert and > and pass through the yaml parser and dtgendoc. The reason is that it's > pure reformatting work. It doesn't take any special knowledge of the > hardware and it doesn't involve reviewing dts files to extra > additional information.
Re: [RFC PATCH 0/5] DT binding documents using text markup
On Thu, Aug 27, 2015 at 10:23 PM, Matt Porter wrote: > During the Device Tree microconference at Linux Plumbers 2015, we had > a short discussion about how to improve DT Binding Documentation. A > number of issues were raised (again, as these things have been > discussed in the past) including: > > * Inconsistency between binding documents due to prose text > format. > * Inability to reliably machine read bindings for mass update > or search. > * Bit rot of bindings as new conventions are agreed upon but > only new bindings are changed. > > Grant Likely probably summed up the issue best with "...as long as > bindings are human readable, we'll have issues...". I'd rephrase that to "... as long as bindings are not machine readable, we'll have issues...". Human readable and machine readable are not mutually exclusive. XML is not, IMHO, human readable, but YAML looks pretty good, so I think it's a good choice. > The context > of that comment was, of course, regarding our current documents > written in very inconsistent prose style. When the topic of needing > the bindings in a rigid format was raised, there was general head > nodding that this was needed. It was noted that this has been > discussed many times before and nothing has been done. > > My proposed solution to the problem is to convert all DT bindings > a rigid text markup format. In choosing a text markup language my > requirements were: > > 1) Human readable > 2) Well documented > 3) Easy to translate to other data formats > 4) Well supported by tools and libraries > > After looking at a number of markup options, YAML stood out as the > one that meets all of these requirements. The YAML syntax is adopted > in many projects specifically because of the high level of readability. > A comprehensive spec is at http://www.yaml.org/spec/1.2/spec.html. > There's a number of tools to convert between YAML and other popular > data formats such as JSON and XML. XML was cited by Behan Webster > during the microconference as an important data format as the type > of developers that may produce comprehensive DTS Binding validation > tools will want to use XML. Every major scripting language has a > high level binding to the low level libyaml C library to facilitate > handling of YAML data files. > > One caveat with YAML is it does not tolerate tabs. Yes, I said it. > No tabs! This can be managed with proper editor modes and also with > helper scripts to strip tabs to aid in people passing planned > checkpatch.pl checks that would run YAML DT Binding specific tag > validators for new bindings. This will be an endless source of pain. Based on your examples, it looks like we only need a very small subset of YAML. I'd be willing to write a parser for the language, but we might lose the ability to do conversions to other formats. It depends on how comprehensive and automatic the helper scripts to strip tabs would be. > > The scope of the initial YAML DT Binding format was specifically > limited to supporting *only* the content we have in bindings today. > The idea here is to propose and agree on something that will take > us just a few steps in the right direction. If we move *all* current > binding content to a machine parseable format, additional features > can be added with more automation and scripting. As it stands today, > because of the inconsistency of the wording of the files, we can't > add a lot of new features to the content until we convert what we > have today into a standard format. > > With that said, it should be noted that some new features such as > "type" tags to indicate cell types could be added to support > additional DTS validation beyond what the current content supports. > Another possibility is adding "range" type information to validate > the legal values for a cell. I would recommend we add type and something for value constraints. Even if we don't add populate those immediately, I think thinking through some of the possible uses will be helpful to avoid doing something dumb. 'choices' or 'enums' seem frequently used. I see lots of "must be one of..." in the binding docs I read. Another important thing to document would be a 'superclass' or 'inherit-from' attribute, for things that are of a specific type (i2c, regulator, clock, etc.) > This series is broken up into three major parts: > > 1) The documentation defining the YAML DT binding format > 2) A skeleton device binding example illustrating use of this format > 3) Some real binding conversions (eeprom.txt, phy-bindings.txt, and >ti-phy.txt > > As a proof of concept of what can be done with a proper machine > readable DT binding source file, there's a simple markdown document > generator at https://github.com/konsulko/dtgendoc. Also, to see > actual output from the generator, the generated markdown from those > bindings is viewable at https://github.com/konsulko/dtgendoc/wiki > > There
Re: [RFC PATCH 0/5] DT binding documents using text markup
On Fri, Aug 28, 2015 at 12:13 PM, Matt Porter wrote: > On Fri, Aug 28, 2015 at 09:26:17AM -0500, Rob Herring wrote: >> On Fri, Aug 28, 2015 at 12:23 AM, Matt Porter wrote: >> > During the Device Tree microconference at Linux Plumbers 2015, we had >> > a short discussion about how to improve DT Binding Documentation. A [...] >> > One caveat with YAML is it does not tolerate tabs. Yes, I said it. >> > No tabs! This can be managed with proper editor modes and also with >> > helper scripts to strip tabs to aid in people passing planned >> > checkpatch.pl checks that would run YAML DT Binding specific tag >> > validators for new bindings. >> >> What do parsers do with tabs? Throw an error? > > Yes, they throw an error. Keep in mind that most of what I used to start > are general purpose conversion tools on top of a particular scripting > languages's high level binding to libyaml. The error output leaves a > bit to be desired for our use case. In any case, when I was developing > the skeleton.yaml I used the yaml script from > https://github.com/ryo/yamltools to catch all these syntax errors I > was inserting..like tabs. the PyYaml binding being used in my PoC > dtgendoc does the same thing but I don't gracefully handle those > errors like we could. > >> Beyond tabs, how do we check files can be parsed both generically and >> for any binding specific requirements. We now need a schema for >> checking the schema. We need some equivalent to compile testing. > > Right. So, I think what you are touching on is something I should > have expanded on in the TODO list. Basically, we need a scripted > tool that we run from checkpatch.pl that 1) reads the .yaml and > validates the YAML itself (that comes for free in the high level > parsers) reporting errors in a sensical manner 2) validates our > DT binding specific tags We all know that no one runs checkpatch.pl. ;) I really want the basic checking of the doc files to run from make (and run by 0-day). The tool dependency could be an issue though. However, DocBook builds from make and I don't think many people check that regularly. Then there is using the binding docs to check dts files. That should probably be part of the dtb building. > Now, I would caution about trying to do too much on Day 1 or we > could end up back at the "never doing anything" stage. Certainly, but I would like to have a plan for what Day 2 and 3 look like. > It would > be an improvement to simply check that the basic tags exist as > shown in the [R] or [O] fields in the documentation. One thing > I should point out is that I carefully avoided marking some tags > as [R] where existing bindings don't have them...even if logically, > a description should be required on every binding. The idea here > is to avoid updating content at the same time that we are updating > the format. Rather, I think it would be better to get the base > format updated, then come back with a janitorial team and add > descriptions (since now we can generate a worklist of those > bindings missing a top-level description) and systematically > fix those and review with the appropriate maintainers. Yes. Any checking would be a great improvement. [...] >> > When we decide on a text markup format that is acceptable, then the >> > next step is to convert all the bindings. That process would start >> > with the complete set of generic bindings as they will be referenced >> > by the actual device bindings. >> >> You are going to do that for everyone, right? ;) > > Let's just say that I'm banking on others helping here once we have > a format agreed upon. If we can hold the binding doc schema definition > initially to just define tags for content that already exists in our > textual binding docs, the effort for conversion is tolerable. To give > an example, that phy-bindings.txt, it took 15 minutes to convert and > and pass through the yaml parser and dtgendoc. The reason is that it's > pure reformatting work. It doesn't take any special knowledge of the > hardware and it doesn't involve reviewing dts files to extra > additional information. Some of the annoyances can be streamlined > like tab stripping and handling the two space indentation to make > this process faster. One of my next things is to get a simple tool > going that reports problems with conversions, essentially what I > said was needed to integrate with checkpatch, so this process of > conversion is even faster. Trivial peripheral bindings like eeprom.txt > can be done in 5 minutes or so right now. What if instead of using the docs as a starting point, we use dts files as a starting point? They give us something parse-able that we can do some automation with. Additionaly, perhaps we can do a mass conversion of all the doc files from txt to yaml where all the current text is converted to comments and we fill in boilerplate and whatever we can convert with some automation? The downside here would be it will be hard to tell which conversions are complete. Put
Re: [RFC PATCH 0/5] DT binding documents using text markup
On 8/27/2015 10:23 PM, Matt Porter wrote: > During the Device Tree microconference at Linux Plumbers 2015, we had > a short discussion about how to improve DT Binding Documentation. A > number of issues were raised (again, as these things have been > discussed in the past) including: > > * Inconsistency between binding documents due to prose text > format. > * Inability to reliably machine read bindings for mass update > or search. > * Bit rot of bindings as new conventions are agreed upon but > only new bindings are changed. > > Grant Likely probably summed up the issue best with "...as long as > bindings are human readable, we'll have issues...". The context > of that comment was, of course, regarding our current documents > written in very inconsistent prose style. When the topic of needing > the bindings in a rigid format was raised, there was general head > nodding that this was needed. It was noted that this has been > discussed many times before and nothing has been done. > > My proposed solution to the problem is to convert all DT bindings > a rigid text markup format. In choosing a text markup language my > requirements were: > > 1) Human readable > 2) Well documented > 3) Easy to translate to other data formats > 4) Well supported by tools and libraries Thanks for taking this on! The proposal looks like a great start. The requirements look good. > After looking at a number of markup options, YAML stood out as the > one that meets all of these requirements. The YAML syntax is adopted > in many projects specifically because of the high level of readability. > A comprehensive spec is at http://www.yaml.org/spec/1.2/spec.html. > There's a number of tools to convert between YAML and other popular > data formats such as JSON and XML. XML was cited by Behan Webster > during the microconference as an important data format as the type > of developers that may produce comprehensive DTS Binding validation > tools will want to use XML. Every major scripting language has a > high level binding to the low level libyaml C library to facilitate > handling of YAML data files. > > One caveat with YAML is it does not tolerate tabs. Yes, I said it. > No tabs! This can be managed with proper editor modes and also with > helper scripts to strip tabs to aid in people passing planned > checkpatch.pl checks that would run YAML DT Binding specific tag > validators for new bindings. I'm just a casual user of some markup languages, nowhere near knowledgeable or an expert, so will be listening to reasoned opinions on the choice of YAML. But a quick look at the spec you referenced and the examples you provided gives me some confidence that YAML will work. > The scope of the initial YAML DT Binding format was specifically > limited to supporting *only* the content we have in bindings today. > The idea here is to propose and agree on something that will take > us just a few steps in the right direction. If we move *all* current Yes, this is a great approach so we can start making reasonable progress. (Caveat below.) > binding content to a machine parseable format, additional features > can be added with more automation and scripting. As it stands today, > because of the inconsistency of the wording of the files, we can't > add a lot of new features to the content until we convert what we > have today into a standard format. > > With that said, it should be noted that some new features such as > "type" tags to indicate cell types could be added to support > additional DTS validation beyond what the current content supports. > Another possibility is adding "range" type information to validate > the legal values for a cell. The caveat: I think we should look at some obvious features (such as "type" that you mention) to make sure that YAML and the chosen structuring will easily and effectively support the features. Thinking about how some features might be implemented led to comments that I will make in replies to the other patches. I agree that we should not attempt to add the additional information enabled by the new features if the information is not in the present bindings. Adding the new information in a subsequent pass seems to be a good strategy. > > This series is broken up into three major parts: > > 1) The documentation defining the YAML DT binding format > 2) A skeleton device binding example illustrating use of this format > 3) Some real binding conversions (eeprom.txt, phy-bindings.txt, and >ti-phy.txt > > As a proof of concept of what can be done with a proper machine > readable DT binding source file, there's a simple markdown document > generator at https://github.com/konsulko/dtgendoc. Also, to see > actual output from the generator, the generated markdown from those > bindings is viewable at https://github.com/konsulko/dtgendoc/wiki > > There's a lot of other possibilities for validation tools using > only the data w
Re: [RFC PATCH 0/5] DT binding documents using text markup
On Fri, Aug 28, 2015 at 09:26:17AM -0500, Rob Herring wrote: > On Fri, Aug 28, 2015 at 12:23 AM, Matt Porter wrote: > > During the Device Tree microconference at Linux Plumbers 2015, we had > > a short discussion about how to improve DT Binding Documentation. A > > number of issues were raised (again, as these things have been > > discussed in the past) including: > > > > * Inconsistency between binding documents due to prose text > > format. > > * Inability to reliably machine read bindings for mass update > > or search. > > * Bit rot of bindings as new conventions are agreed upon but > > only new bindings are changed. > > Thanks for pulling this together. > > > Grant Likely probably summed up the issue best with "...as long as > > bindings are human readable, we'll have issues...". The context > > of that comment was, of course, regarding our current documents > > written in very inconsistent prose style. When the topic of needing > > the bindings in a rigid format was raised, there was general head > > nodding that this was needed. It was noted that this has been > > discussed many times before and nothing has been done. > > > > My proposed solution to the problem is to convert all DT bindings > > a rigid text markup format. In choosing a text markup language my > > requirements were: > > > > 1) Human readable > > 2) Well documented > > 3) Easy to translate to other data formats > > 4) Well supported by tools and libraries > > > > After looking at a number of markup options, YAML stood out as the > > one that meets all of these requirements. The YAML syntax is adopted > > in many projects specifically because of the high level of readability. > > A comprehensive spec is at http://www.yaml.org/spec/1.2/spec.html. > > There's a number of tools to convert between YAML and other popular > > data formats such as JSON and XML. XML was cited by Behan Webster > > during the microconference as an important data format as the type > > of developers that may produce comprehensive DTS Binding validation > > tools will want to use XML. Every major scripting language has a > > high level binding to the low level libyaml C library to facilitate > > handling of YAML data files. > > Being markup language novice, this looks good to me. > > > One caveat with YAML is it does not tolerate tabs. Yes, I said it. > > No tabs! This can be managed with proper editor modes and also with > > helper scripts to strip tabs to aid in people passing planned > > checkpatch.pl checks that would run YAML DT Binding specific tag > > validators for new bindings. > > What do parsers do with tabs? Throw an error? Yes, they throw an error. Keep in mind that most of what I used to start are general purpose conversion tools on top of a particular scripting languages's high level binding to libyaml. The error output leaves a bit to be desired for our use case. In any case, when I was developing the skeleton.yaml I used the yaml script from https://github.com/ryo/yamltools to catch all these syntax errors I was inserting..like tabs. the PyYaml binding being used in my PoC dtgendoc does the same thing but I don't gracefully handle those errors like we could. > Beyond tabs, how do we check files can be parsed both generically and > for any binding specific requirements. We now need a schema for > checking the schema. We need some equivalent to compile testing. Right. So, I think what you are touching on is something I should have expanded on in the TODO list. Basically, we need a scripted tool that we run from checkpatch.pl that 1) reads the .yaml and validates the YAML itself (that comes for free in the high level parsers) reporting errors in a sensical manner 2) validates our DT binding specific tags Now, I would caution about trying to do too much on Day 1 or we could end up back at the "never doing anything" stage. It would be an improvement to simply check that the basic tags exist as shown in the [R] or [O] fields in the documentation. One thing I should point out is that I carefully avoided marking some tags as [R] where existing bindings don't have them...even if logically, a description should be required on every binding. The idea here is to avoid updating content at the same time that we are updating the format. Rather, I think it would be better to get the base format updated, then come back with a janitorial team and add descriptions (since now we can generate a worklist of those bindings missing a top-level description) and systematically fix those and review with the appropriate maintainers. > An example such as checking that compatible strings are documented as > checkpatch.pl does would be nice. Roughly, that would be just list all > compatible values. Ok, so my comments above were strictly about a validator for the binding doc submission itself. I can add an example based on your checkpatch.pl to adapt it to the .yaml compatible tags.
Re: [RFC PATCH 0/5] DT binding documents using text markup
On Fri, Aug 28, 2015 at 12:23 AM, Matt Porter wrote: > During the Device Tree microconference at Linux Plumbers 2015, we had > a short discussion about how to improve DT Binding Documentation. A > number of issues were raised (again, as these things have been > discussed in the past) including: > > * Inconsistency between binding documents due to prose text > format. > * Inability to reliably machine read bindings for mass update > or search. > * Bit rot of bindings as new conventions are agreed upon but > only new bindings are changed. Thanks for pulling this together. > Grant Likely probably summed up the issue best with "...as long as > bindings are human readable, we'll have issues...". The context > of that comment was, of course, regarding our current documents > written in very inconsistent prose style. When the topic of needing > the bindings in a rigid format was raised, there was general head > nodding that this was needed. It was noted that this has been > discussed many times before and nothing has been done. > > My proposed solution to the problem is to convert all DT bindings > a rigid text markup format. In choosing a text markup language my > requirements were: > > 1) Human readable > 2) Well documented > 3) Easy to translate to other data formats > 4) Well supported by tools and libraries > > After looking at a number of markup options, YAML stood out as the > one that meets all of these requirements. The YAML syntax is adopted > in many projects specifically because of the high level of readability. > A comprehensive spec is at http://www.yaml.org/spec/1.2/spec.html. > There's a number of tools to convert between YAML and other popular > data formats such as JSON and XML. XML was cited by Behan Webster > during the microconference as an important data format as the type > of developers that may produce comprehensive DTS Binding validation > tools will want to use XML. Every major scripting language has a > high level binding to the low level libyaml C library to facilitate > handling of YAML data files. Being markup language novice, this looks good to me. > One caveat with YAML is it does not tolerate tabs. Yes, I said it. > No tabs! This can be managed with proper editor modes and also with > helper scripts to strip tabs to aid in people passing planned > checkpatch.pl checks that would run YAML DT Binding specific tag > validators for new bindings. What do parsers do with tabs? Throw an error? Beyond tabs, how do we check files can be parsed both generically and for any binding specific requirements. We now need a schema for checking the schema. We need some equivalent to compile testing. An example such as checking that compatible strings are documented as checkpatch.pl does would be nice. Roughly, that would be just list all compatible values. > The scope of the initial YAML DT Binding format was specifically > limited to supporting *only* the content we have in bindings today. > The idea here is to propose and agree on something that will take > us just a few steps in the right direction. If we move *all* current > binding content to a machine parseable format, additional features > can be added with more automation and scripting. As it stands today, > because of the inconsistency of the wording of the files, we can't > add a lot of new features to the content until we convert what we > have today into a standard format. > > With that said, it should be noted that some new features such as > "type" tags to indicate cell types could be added to support > additional DTS validation beyond what the current content supports. > Another possibility is adding "range" type information to validate > the legal values for a cell. > > This series is broken up into three major parts: > > 1) The documentation defining the YAML DT binding format > 2) A skeleton device binding example illustrating use of this format > 3) Some real binding conversions (eeprom.txt, phy-bindings.txt, and >ti-phy.txt > > As a proof of concept of what can be done with a proper machine > readable DT binding source file, there's a simple markdown document > generator at https://github.com/konsulko/dtgendoc. Also, to see > actual output from the generator, the generated markdown from those > bindings is viewable at https://github.com/konsulko/dtgendoc/wiki Nice. > There's a lot of other possibilities for validation tools using > only the data we have today in the bindings. In addition, Frank > Rowand covered some DT debug techniques that would benefit from > the binding documentation being 100% reliably searchable. > > I found it useful to see a side-by-side view of a converted doc > versus the original content, so here's a screenshot of eeprom.txt > vs. eeprom.yaml: > https://github.com/konsulko/dtgendoc/wiki#eepromtxt-vs-eepromyaml > > When we decide on a text markup format that is acceptable, then the > next step is to con
[RFC PATCH 0/5] DT binding documents using text markup
During the Device Tree microconference at Linux Plumbers 2015, we had a short discussion about how to improve DT Binding Documentation. A number of issues were raised (again, as these things have been discussed in the past) including: * Inconsistency between binding documents due to prose text format. * Inability to reliably machine read bindings for mass update or search. * Bit rot of bindings as new conventions are agreed upon but only new bindings are changed. Grant Likely probably summed up the issue best with "...as long as bindings are human readable, we'll have issues...". The context of that comment was, of course, regarding our current documents written in very inconsistent prose style. When the topic of needing the bindings in a rigid format was raised, there was general head nodding that this was needed. It was noted that this has been discussed many times before and nothing has been done. My proposed solution to the problem is to convert all DT bindings a rigid text markup format. In choosing a text markup language my requirements were: 1) Human readable 2) Well documented 3) Easy to translate to other data formats 4) Well supported by tools and libraries After looking at a number of markup options, YAML stood out as the one that meets all of these requirements. The YAML syntax is adopted in many projects specifically because of the high level of readability. A comprehensive spec is at http://www.yaml.org/spec/1.2/spec.html. There's a number of tools to convert between YAML and other popular data formats such as JSON and XML. XML was cited by Behan Webster during the microconference as an important data format as the type of developers that may produce comprehensive DTS Binding validation tools will want to use XML. Every major scripting language has a high level binding to the low level libyaml C library to facilitate handling of YAML data files. One caveat with YAML is it does not tolerate tabs. Yes, I said it. No tabs! This can be managed with proper editor modes and also with helper scripts to strip tabs to aid in people passing planned checkpatch.pl checks that would run YAML DT Binding specific tag validators for new bindings. The scope of the initial YAML DT Binding format was specifically limited to supporting *only* the content we have in bindings today. The idea here is to propose and agree on something that will take us just a few steps in the right direction. If we move *all* current binding content to a machine parseable format, additional features can be added with more automation and scripting. As it stands today, because of the inconsistency of the wording of the files, we can't add a lot of new features to the content until we convert what we have today into a standard format. With that said, it should be noted that some new features such as "type" tags to indicate cell types could be added to support additional DTS validation beyond what the current content supports. Another possibility is adding "range" type information to validate the legal values for a cell. This series is broken up into three major parts: 1) The documentation defining the YAML DT binding format 2) A skeleton device binding example illustrating use of this format 3) Some real binding conversions (eeprom.txt, phy-bindings.txt, and ti-phy.txt As a proof of concept of what can be done with a proper machine readable DT binding source file, there's a simple markdown document generator at https://github.com/konsulko/dtgendoc. Also, to see actual output from the generator, the generated markdown from those bindings is viewable at https://github.com/konsulko/dtgendoc/wiki There's a lot of other possibilities for validation tools using only the data we have today in the bindings. In addition, Frank Rowand covered some DT debug techniques that would benefit from the binding documentation being 100% reliably searchable. I found it useful to see a side-by-side view of a converted doc versus the original content, so here's a screenshot of eeprom.txt vs. eeprom.yaml: https://github.com/konsulko/dtgendoc/wiki#eepromtxt-vs-eepromyaml When we decide on a text markup format that is acceptable, then the next step is to convert all the bindings. That process would start with the complete set of generic bindings as they will be referenced by the actual device bindings. If the RFC wasn't explicit enough...comments welcome. -Matt Matt Porter (5): Documentation: dt-bindings: add documentation on new DT binding format Documentation: dt-bindings: add example DT binding document Documentation: dt-bindings: add YAML eeprom binding Documentation: dt-bindings: phy: add YAML generic PHY binding Documentation: dt-bindings: phy: add YAML TI PHY binding .../devicetree/bindings/dt-binding-format.txt | 106 + Documentation/devicetree/bindings/eeprom.yaml | 44 ++ .../devicetree/bindings/phy/phy-bindi
