Re: [RFC PATCH 0/5] DT binding documents using text markup

2015-09-08 Thread Rob Herring 
On Tue, Sep 8, 2015 at 9:36 AM, Matt Porter  wrote:
> On Tue, Sep 01, 2015 at 01:24:17PM -0500, Rob Herring wrote:
>> On Tue, Sep 1, 2015 at 1:03 PM, Tim Bird  wrote:
>> > On Tue, Sep 1, 2015 at 10:35 AM, Frank Rowand  
>> > wrote:
>> >> On 9/1/2015 10:14 AM, Tim Bird wrote:
>> >>> On Fri, Aug 28, 2015 at 10:13 AM, Matt Porter  
>> >>> wrote:
>> >>
>> >> < snip >
>> >>
>> 
>>  But to answer your question, if we get a format I'll do
>>  conversions and hope I'm not alone.
>> >>>
>> >>> I'm sure others will help out.  I will, and I'm pretty sure we can get
>> >>> some conversion sprints set up at conferences (I know I can set aside
>> >>> some time or resources at ELC in the spring - it might be too late for
>> >>> ELCE in October to set up a scheduled block of time, but we can start
>> >>> getting the word out.)  As I said in my other e-mail, one doesn't have
>> >>> to be a kernel coder to do this, and the conversions should be pretty
>> >>> straight-forward.
>> >>>  -- Tim
>> >>>
>> >>
>> >> A conversion sprint at ELCE sounds like a good idea if we can find a
>> >> good time to schedule it.  I can help, so there will be at least two
>> >> of us who can help people as they encounter issues.
>> >
>> > Even if we don't find a block of time, we can do something like
>> > announce a "contest", ask people to do something in their spare time,
>> > and find some way to get them a raffle ticket if they submit a patch
>> > with a conversion.  Then hold an extra prize drawing during the
>> > closing session, with just those raffle tickets, and give someone a
>> > nice award for contributing.  Of course, there's always the adage that
>> > you should be careful what you measure and reward...  We don't want a
>> > flood of crappy conversions, with a time constraint on the review.
>> > I'll think some more about this.  An alternative would be to have a
>> > contest announced ahead of the event, with enough time for people to
>> > submit and get reviewed.
>>
>> Sounds like a review nightmare. That's another reason why as much
>> automated conversion we can do, the better.
>
> I don't want to discount the value of interested people getting together
> f2f to review these and potentially clean them up for submission. That
> depends on what we thinking is the minimal "in progress" conversion
> that can be place upstream. i.e. is it simply compatible strings
> autoconverted to tags and the entire current document in comments?

That is sufficient for me (I reserve the right to change my mind).
Logistically, it needs to be a script that can be run before/during
the merge window and perhaps again after. I'd guess the long pole here
is how we validate the .yaml files.

>> > By the way - I presume the new docs will replace the existing ones,
>> > but I could imagine wanting to have them live side-by-side
>> > temporarily.  Any thoughts on this?  Will file name or location
>> > changes be allowed during the conversion?
>>
>> I proposed some ideas earlier in the thread. Either we can have both
>> side by side or do a mass conversion to YAML making the existing doc a
>> comment (add # prefix).
>>
>
> Were you thinking that this automated conversion would be sufficient
> for an initial commit? I'm not sure if I misunderstood in your separate
> comments and was looking at this as something that would be hand edited
> to move the existing doc (# prefix) into description tags where
> appropriate.

Yes, I think it is more important to have infrastructure in place to
enable others than how much is converted initially. Certainly any
changes based on existing docs will have to be manual, but we may be
able to do multiple passes of automated conversions. Sharing any
conversion scripts is important to enable others for that as well.

>> Any renames/moving should be separate (there's some clean-up I'd like
>> to there as well). Exact rules depend on the approach, but we need to
>> be able to tell which bindings conversions are not started, in
>> progress, or complete.
>
> If we add .yaml in place we can identify what's in progress by the fact
> that a .yaml exists with the same filename and then based on which
> tags have been populated (such as type: and constraints: not yet
> populated) then we know the state.

You mean keeping the .txt and .yaml until done? If we are copying in
the current doc, then I'd rather not do that. I don't think looking at
the tags will work as it would be hard to distinguish incomplete from
done. However, we could simply have an "in-progress" tag that is
removed when done (might as well take advantage of our new found
structured docs).

Rob
--
To unsubscribe from this list: send the line "unsubscribe devicetree-spec" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH 0/5] DT binding documents using text markup

2015-09-08 Thread Matt Porter 
On Tue, Sep 01, 2015 at 10:35:30AM -0700, Frank Rowand wrote:
> On 9/1/2015 10:14 AM, Tim Bird wrote:
> > On Fri, Aug 28, 2015 at 10:13 AM, Matt Porter  wrote:
> 
> < snip >
> 
> >>
> >> But to answer your question, if we get a format I'll do
> >> conversions and hope I'm not alone.
> > 
> > I'm sure others will help out.  I will, and I'm pretty sure we can get
> > some conversion sprints set up at conferences (I know I can set aside
> > some time or resources at ELC in the spring - it might be too late for
> > ELCE in October to set up a scheduled block of time, but we can start
> > getting the word out.)  As I said in my other e-mail, one doesn't have
> > to be a kernel coder to do this, and the conversions should be pretty
> > straight-forward.
> >  -- Tim
> > 
> 
> A conversion sprint at ELCE sounds like a good idea if we can find a
> good time to schedule it.  I can help, so there will be at least two
> of us who can help people as they encounter issues.

I'll be there as well.

-Matt
--
To unsubscribe from this list: send the line "unsubscribe devicetree-spec" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[RFC PATCH 0/5] DT binding documents using text markup

2015-08-27 Thread Matt Porter 
During the Device Tree microconference at Linux Plumbers 2015, we had
a short discussion about how to improve DT Binding Documentation. A
number of issues were raised (again, as these things have been
discussed in the past) including:

* Inconsistency between binding documents due to prose text
  format.
* Inability to reliably machine read bindings for mass update
  or search.
* Bit rot of bindings as new conventions are agreed upon but
  only new bindings are changed.

Grant Likely probably summed up the issue best with ...as long as
bindings are human readable, we'll have issues The context
of that comment was, of course, regarding our current documents
written in very inconsistent prose style. When the topic of needing
the bindings in a rigid format was raised, there was general head
nodding that this was needed. It was noted that this has been
discussed many times before and nothing has been done.

My proposed solution to the problem is to convert all DT bindings
a rigid text markup format. In choosing a text markup language my
requirements were:

1) Human readable
2) Well documented
3) Easy to translate to other data formats
4) Well supported by tools and libraries

After looking at a number of markup options, YAML stood out as the
one that meets all of these requirements. The YAML syntax is adopted
in many projects specifically because of the high level of readability.
A comprehensive spec is at http://www.yaml.org/spec/1.2/spec.html.
There's a number of tools to convert between YAML and other popular
data formats such as JSON and XML. XML was cited by Behan Webster
during the microconference as an important data format as the type
of developers that may produce comprehensive DTS Binding validation
tools will want to use XML. Every major scripting language has a
high level binding to the low level libyaml C library to facilitate
handling of YAML data files.

One caveat with YAML is it does not tolerate tabs. Yes, I said it.
No tabs! This can be managed with proper editor modes and also with
helper scripts to strip tabs to aid in people passing planned
checkpatch.pl checks that would run YAML DT Binding specific tag
validators for new bindings.

The scope of the initial YAML DT Binding format was specifically
limited to supporting *only* the content we have in bindings today.
The idea here is to propose and agree on something that will take
us just a few steps in the right direction. If we move *all* current
binding content to a machine parseable format, additional features
can be added with more automation and scripting. As it stands today,
because of the inconsistency of the wording of the files, we can't
add a lot of new features to the content until we convert what we
have today into a standard format.

With that said, it should be noted that some new features such as
type tags to indicate cell types could be added to support
additional DTS validation beyond what the current content supports.
Another possibility is adding range type information to validate
the legal values for a cell.

This series is broken up into three major parts:

1) The documentation defining the YAML DT binding format
2) A skeleton device binding example illustrating use of this format
3) Some real binding conversions (eeprom.txt, phy-bindings.txt, and
   ti-phy.txt

As a proof of concept of what can be done with a proper machine
readable DT binding source file, there's a simple markdown document
generator at https://github.com/konsulko/dtgendoc. Also, to see
actual output from the generator, the generated markdown from those
bindings is viewable at https://github.com/konsulko/dtgendoc/wiki

There's a lot of other possibilities for validation tools using
only the data we have today in the bindings. In addition, Frank
Rowand covered some DT debug techniques that would benefit from
the binding documentation being 100% reliably searchable.

I found it useful to see a side-by-side view of a converted doc
versus the original content, so here's a screenshot of eeprom.txt
vs. eeprom.yaml:
https://github.com/konsulko/dtgendoc/wiki#eepromtxt-vs-eepromyaml

When we decide on a text markup format that is acceptable, then the
next step is to convert all the bindings. That process would start
with the complete set of generic bindings as they will be referenced
by the actual device bindings.

If the RFC wasn't explicit enough...comments welcome.

-Matt

Matt Porter (5):
  Documentation: dt-bindings: add documentation on new DT binding format
  Documentation: dt-bindings: add example DT binding document
  Documentation: dt-bindings: add YAML eeprom binding
  Documentation: dt-bindings: phy: add YAML generic PHY binding
  Documentation: dt-bindings: phy: add YAML TI PHY binding

 .../devicetree/bindings/dt-binding-format.txt  | 106 +
 Documentation/devicetree/bindings/eeprom.yaml  |  44 ++