All, I tweaked the layout of the generated documentation and put a new rendered copy here: http://danbress.github.io/generated-documentation/components-02072015/
Please provide any feedback you may have. I changed the css for the table a little bit to make things more readable. I'm not married to it, so if anyone wants to build upon it, please go forward. I am cancelling my old pull request and issuing a new one as I did the changes for this in a new branch. Let me know how this looks. Dan Bress Software Engineer ONYX Consulting Services ________________________________________ From: Daniel Bress <[email protected]> Sent: Sunday, January 25, 2015 9:43 AM To: [email protected] Subject: Re: Processor documentation plugin Jenn, Good point, your text makes a lot more sense than what I had. Thanks for that! I just included it and regenerated the docs, let me know what you think. Dan Bress Software Engineer ONYX Consulting Services ________________________________________ From: Jennifer Barnabee <[email protected]> Sent: Sunday, January 25, 2015 9:15 AM To: [email protected] Subject: Re: Processor documentation plugin Dan, It looks good. Assuming we decide to move forward with it, we'll want to change the boilerplate paragraph under Properties to suit what you have a little better. I propose something like the following, but it may need to change as things evolve. Please see what you think... In the list below, the names of required properties appear in *bold*. Any other properties (not in bold) are considered optional. The table also indicates any default values, whether a property supports the NiFi Expression Language (or simply EL), and whether a property is considered "sensitive", meaning that its value will be encrypted. Before entering a value in a sensitive property, ensure that the *nifi.properties* file has an entry for the property *nifi.sensitive.props.key.* -Jenn On Sun, Jan 25, 2015 at 9:01 AM, Daniel Bress <[email protected]> wrote: > Tony, > Thanks for pointing that out. I had a "\" when I needed a "/". I > fixed this and re uploaded it. Let me know how that looks. > > Dan Bress > Software Engineer > ONYX Consulting Services > > ________________________________________ > From: Tony Kurc <[email protected]> > Sent: Saturday, January 24, 2015 8:07 PM > To: [email protected] > Subject: Re: Processor documentation plugin > > Dan, > For whatever reason, when I follow the hyperlinks, the last "slash" is a \ > instead of a / and I get a 404 ( > > http://danbress.github.io/generated-documentation/components/org.apache.nifi.processors.standard.InvokeHTTP%5Cindex.html > ). > > > > > On Sat, Jan 24, 2015 at 1:01 PM, Daniel Bress <[email protected]> > wrote: > > > All, > > I've generated the first round of documentation I'd like everyone to > > look at to see if they like this new format. Its really only slightly > > changed from the current format(using tables instead of lists for > > Properties and Relationships), but its all machine generated, except for > > the stuff linked by "Additional Details". > > > > Again, this would be generated by the app at startup. I'd also like > to > > have this generated sometime during the build and linked from the > website. > > For now I just pulled that generated html and uploaded it to my github > page > > for your review. > > > > http://danbress.github.io/generated-documentation/components/ > > > > Let me know what you think! I'm open to any ideas/suggestions. > > > > Dan Bress > > Software Engineer > > ONYX Consulting Services > > > > ________________________________________ > > From: Mark Payne <[email protected]> > > Sent: Thursday, January 22, 2015 8:16 AM > > To: [email protected] > > Subject: RE: Processor documentation plugin > > > > Dan, > > OK, now I understand where you're coming from. Sorry, I misunderstood the > > idea before. If you think it would be too verbose then I think we should > > hold off for now. I think it's best to keep it simple. If we want to add > it > > in later, we can. If we add it now and decide it's too verbose, it would > be > > awkward to go back and remove documentation later but it's easy to later > > add in something that's missing. > > > > So in short I agree with your assessment :) > > -Mark > > > > > From: [email protected] > > > To: [email protected] > > > Subject: Re: Processor documentation plugin > > > Date: Thu, 22 Jan 2015 03:13:16 +0000 > > > > > > Mark, > > > No worries. On the dynamic property tip, I see what you are saying. > > I was hoping to provide the same level of detail in describing dynamic > > properties as non-dynamic ones, but right now I see no way to do that. > > Thats fine, I'll put that part(providing detailed documentation of the > > dynamic properties) on the back burner. > > > > > > On the Validator tip, for each PropertyDescriptor I am getting the > > list of AllowableValues so that I can say "This property has X, Y and Z > for > > valid values". If there are no AllowableValues present I would like to > > look at the validators and say "This property requires a positive > > integer". or "This requires a <data size> followed by a <data unit>". > > Although typing this all out now makes it seem like it might be too > verbose > > and cluttered. I think I'll forget this idea and just work with > > AllowableValues. > > > > > > Thanks again, > > > > > > Dan Bress > > > Software Engineer > > > ONYX Consulting Services > > > > > > ________________________________________ > > > From: Mark Payne <[email protected]> > > > Sent: Tuesday, January 20, 2015 12:16 PM > > > To: [email protected] > > > Subject: RE: Processor documentation plugin > > > > > > Dan, > > > Sorry, I never did respond to this e-mail. > > > You cannot get the dynamic PropertyDescriptor because the > > AbstractConfigurableComponent (the base abstract class from which every > > processor, controller service, reporting task that I know of extends) > > provides the following method signature: > > > protected PropertyDescriptor > getSupportedDynamicPropertyDescriptor(final > > String propertyDescriptorName) > > > I.e., there is not a PropertyDescriptor for dynamic properties but > > rather the descriptor can be different for every dynamic property. For > > example, consider a Processor that allows dynamic properties but for > every > > property X there must be 2 other properties: X.format (any string) and > > X.size (a non-zero integer). If there were a single PropertyDescriptor, > > this wouldn't be possible because all dynamic properties would have the > > same validators, etc. But by making a property descriptor based on the > > property name, we can easily achieve this. > > > Regarding the second note: I don't think Validator can provide > > information about what it is validating because it doesn't know. For > > instance, PositiveInteger validator just validates that some value is a > > positive integer. It does not (and I think should not) know the context > of > > the value. > > > If anything here doesn't make sense or if you think it could be done > > better, let me know and we can discuss further. > > > Thanks-Mark > > > > > > > From: [email protected] > > > > To: [email protected] > > > > Subject: Re: Processor documentation plugin > > > > Date: Fri, 16 Jan 2015 21:14:41 +0000 > > > > > > > > Mark & Matt, > > > > Cool. While I like the idea of @SupportsDynamicProperties, > > especially if it has a description, I'd also like a guaranteed way to get > > the dynamic PropertyDescriptor so I can interrogate it further. I'd be > > interested in knowing whether it supports expression language, and/or how > > the value of the property is validated. > > > > > > > > On that note, I was thinking it may be helpful to add either an > > annotation or a method to the Validator interface that could provide a > text > > description about what it is validating. Is this useful or overkill? > > > > > > > > Dan Bress > > > > Software Engineer > > > > ONYX Consulting Services > > > > > > > > ________________________________________ > > > > From: Mark Payne <[email protected]> > > > > Sent: Friday, January 16, 2015 11:14 AM > > > > To: [email protected] > > > > Subject: RE: Processor documentation plugin > > > > > > > > It might make sense to have an annotation for this: > > > > @SupportsDynamicProperties > > > > Perhaps even allow developer to provide a string for how they are > used: > > > > @SupportsDynamicProperties("Name of dynamic property becomes > attribute > > name; value becomes attribute value.") > > > > I've begun ticket #6: https://issues.apache.org/jira/browse/NIFI-6 > > but I have not pushed anything to a branch or anything yet. Basically, I > > deprecated the existing annotations and created 3 new packages for them. > > Instead of org.apache.nifi.processor.annotation, we will have: > > > > > > > org.apache.nifi.annotation.documentationorg.apache.nifi.annotation.behaviororg.apache.nifi.annotation.lifecycle > > > > So the @SupportsDynamicProperties could go under 'behavior' if that > > makes sense. > > > > > > > > > Date: Fri, 16 Jan 2015 11:09:28 -0500 > > > > > Subject: Re: Processor documentation plugin > > > > > From: [email protected] > > > > > To: [email protected] > > > > > > > > > > Dan, > > > > > > > > > > We discussed item 1 awhile back and even created a ticket for it. > > However, > > > > > there was always higher priority work that took precedence. I am > not > > sure > > > > > if that ticket was ever created in JIRA for Apache NiFi. If not, we > > should. > > > > > I believe the current mechanism allows the processor to decide if a > > given > > > > > dynamic property should be a supported. I think we would need a > more > > > > > general indicator that dynamic properties are supported at all to > > trigger > > > > > that button state. This would likely help with item 2 as well. > > > > > > > > > > Matt > > > > > > > > > > On Fri, Jan 16, 2015 at 10:45 AM, Daniel Bress < > > [email protected]> > > > > > wrote: > > > > > > > > > > > Mark, > > > > > > Sounds good. I'm on board. I was thinking of having the > > > > > > ControllerServices/ReportingTask's documentation include an XML > > template, > > > > > > but if those will be configurable via the UI then that will not > be > > needed. > > > > > > I'm on board with your rev 1 / rev 2 suggestions too. I'll > get > > to work > > > > > > accordingly. > > > > > > > > > > > > > > > > > > Matt/Mark, > > > > > > Two UI/API questions: > > > > > > 1) If a processor does not support dynamic properties, why does > > the UI let > > > > > > them hit [+] New Property? > > > > > > 2) If I have an instance of Processor class, I can call > > > > > > getSupportedProperties() on it to get a list of the > > required/optional > > > > > > properties. But I don't have a way to know if that processor > > supports > > > > > > dynamic properties or not. Supporting dynamic properties or not > > was > > > > > > something I was planning on reflecting in the documentation. Is > > there any > > > > > > way to detect this currently? > > > > > > > > > > > > Thanks, > > > > > > Dan Bress > > > > > > Software Engineer > > > > > > ONYX Consulting Services > > > > > > > > > > > > ________________________________________ > > > > > > From: Mark Payne <[email protected]> > > > > > > Sent: Wednesday, January 14, 2015 8:51 PM > > > > > > To: [email protected] > > > > > > Subject: RE: Processor documentation plugin > > > > > > > > > > > > Dan, > > > > > > I think this is great! > > > > > > I would recommend we slate this for the 0.1.0 version, which > > includes > > > > > > putting Controller Services and Reporting Tasks in the UI. But by > > the same > > > > > > token I think that makes your first bulletin for "rev 2" more of > a > > priority > > > > > > for "rev 1" if we can get it there. I could pitch in and help if > > needed. > > > > > > I'd also recommend you put the Capability Description in for Rev > 1 > > as > > > > > > that's arguably the most important part of the docs (which may > > make the > > > > > > Tags very easy at that point??) > > > > > > The rest of what you have listed for "rev 2" i would agree should > > be there. > > > > > > > > > > > > Very excited to see this happening! > > > > > > Thanks-Mark > > > > > > > > > > > > > From: [email protected] > > > > > > > To: [email protected] > > > > > > > Subject: Re: Processor documentation plugin > > > > > > > Date: Thu, 15 Jan 2015 01:39:44 +0000 > > > > > > > > > > > > > > Mark et al, > > > > > > > Ok, I think I hear everyone's concerns and ideas. I will > > challenge > > > > > > myself to come up with a solution that makes everyone happy all > > the time. > > > > > > > > > > > > > > The goals of this effort will be: > > > > > > > - Machine generate as much of the basic(processor > > description, > > > > > > property description, relationship description) info as possible. > > This > > > > > > will make the developers lives better by storing this information > > in one > > > > > > place, and make users lives better by having the "inline" > > documentation > > > > > > match the "usage" documentation. This will also ensure that all > > the > > > > > > documentation is formatted consistently. > > > > > > > - Allow the developer to optionally provide additional > > html and > > > > > > images that would be linked from the machine generated > > documentation > > > > > > > - Provide a way for the documentation to be generated > > statically > > > > > > and linked from the Apache NiFi webpage > > > > > > > > > > > > > > A few other 'nice-to-have' goals(probably for rev 2): > > > > > > > - Support documenting ControllerServices and > > ReportingTasks is a > > > > > > similar way > > > > > > > -Maybe leverage existing annotations on these types? > > E.g > > > > > > @Tags, @CapabilityDescription... > > > > > > > - Provide new 'documentation' annotations that would > allow > > a > > > > > > developer to describe the FlowFile attributes a processor uses on > > input, > > > > > > and which attributes a processor modifies on output > > > > > > > - Provide new 'documentation' annotations that describe > the > > > > > > expected input format of the flow file content > > > > > > > - Provide new 'documentation' annotations that describe > > whether > > > > > > the processor modifies the flow file content > > > > > > > - Provide new 'documentation' annotations that describe > > the format > > > > > > of the outbound flow file content > > > > > > > - Provide new 'documentation' annotations that describe > > other NiFi > > > > > > artifacts(processors, controller services, reporting tasks) that > > are > > > > > > related to this one, e.g. "See Also ..." > > > > > > > - Consider if/how current Processor annotations should be > > > > > > displayed (@SideEffectFree, @TriggerSerially, > @TriggerWhenEmpty..., > > > > > > @EventDriven) > > > > > > > > > > > > > > > > > > > > > High level implementation plan: > > > > > > > - Develop a utility class to read an > AbstractProcessor(and > > > > > > existing HTML documention) and generated the basic documentation > > with links > > > > > > to existing HTML > > > > > > > - Integrate this class into the framework such that all > > > > > > documentation displayed within a NiFi instance is generated by > the > > utility > > > > > > > - Integrate this utility into something that can take a > > NiFi > > > > > > build, dig through the nars, find the > > > > > > Processors/ControllerServices/ReportingTasks and generate their > > appropriate > > > > > > HTML documentation so that it can be stored somewhere and linked > > from the > > > > > > Apache NiFi website. > > > > > > > > > > > > > > If this feels like the wrong path, or if you have any other > > ideas > > > > > > please speak up. Thanks! > > > > > > > > > > > > > > Dan Bress > > > > > > > Software Engineer > > > > > > > ONYX Consulting Services > > > > > > > > > > > > > > ________________________________________ > > > > > > > From: Mark Payne <[email protected]> > > > > > > > Sent: Monday, January 12, 2015 3:40 PM > > > > > > > To: [email protected] > > > > > > > Subject: RE: Processor documentation plugin > > > > > > > > > > > > > > Dan, > > > > > > > Yes, that's what I'm saying. Right now, it's all done manually > > by hand, > > > > > > and it's a pain. The down side i think to doing it via a Maven > > plugin is > > > > > > that if the user wants to modify it, they have to rebuild, pull > the > > > > > > generated html out of the target directory, and then modify it > and > > put it > > > > > > in the src/main/resources/docs. Then, if they change something > > they have to > > > > > > do this again. If it's done on-the-fly, rather than building > > static content > > > > > > with a plugin, that doesn't have to happen. > > > > > > > But you may come up with something much better than I'm > > envisioning :) > > > > > > Having a util to do it is a good call though. Then it could be > > used in a > > > > > > maven plugin, by the app, or wherever is most appropriate. > > > > > > > Thanks-Mark > > > > > > > > > > > > > > > From: [email protected] > > > > > > > > To: [email protected] > > > > > > > > Subject: Re: Processor documentation plugin > > > > > > > > Date: Mon, 12 Jan 2015 20:34:03 +0000 > > > > > > > > > > > > > > > > Mark, > > > > > > > > Sorry.. I missed your reply in the thread. > > > > > > > > > > > > > > > > Are you saying that going forward, you want to auto > > generate the > > > > > > doc and allow the developer to include a "more info" link? I > > wanted to > > > > > > confirm that it doesn't work that way now. > > > > > > > > > > > > > > > > I'll think about this a little more. Might write some > code > > to > > > > > > generate the documentation from an AbstractProcessor class, then > > decide > > > > > > later where to integrate it(maven plugin, or somewhere else) > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > Dan Bress > > > > > > > > Software Engineer > > > > > > > > ONYX Consulting Services > > > > > > > > > > > > > > > > ________________________________________ > > > > > > > > From: Mark Payne <[email protected]> > > > > > > > > Sent: Sunday, January 11, 2015 1:23 PM > > > > > > > > To: [email protected] > > > > > > > > Subject: Re: Processor documentation plugin > > > > > > > > > > > > > > > > Dan, > > > > > > > > > > > > > > > > Matt and I discussed this approach before but ultimately > > decided that > > > > > > we like the idea of just auto generating the docs on demand with > a > > "More > > > > > > Info" type of link that lets you add more in html format if you > > care to do > > > > > > so. > > > > > > > > > > > > > > > > Others' thoughts and ideas are welcome though! > > > > > > > > > > > > > > > > Sent from my iPhone > > > > > > > > > > > > > > > > > On Jan 11, 2015, at 12:32 PM, Daniel Bress < > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > Just pulled down the code out of Git and got everything > up > > and > > > > > > running on Windows. This was pretty simple, and the UI looks > > great under > > > > > > chrome. > > > > > > > > > > > > > > > > > > > > > > > > > > > I was thinking of writing a maven plugin to generate the > > processor > > > > > > usage HTML documentation. > > > > > > > > > > > > > > > > > > > > > > > > > > > Looking through the existing processor documentation, it > > seems > > > > > > like most of this information comes directly from the code in the > > > > > > processors, via @CapabilityDescription, and the > > PropertyDescriptors and > > > > > > Relationships. Then if your processor only requires this level > of > > > > > > documentation in the HTML format, you could tie this plugin to > the > > build, > > > > > > and keep your inline processor documentation in sync with the > HTML > > > > > > documentation. > > > > > > > > > > > > > > > > > > > > > > > > > > > What do you think? > > > > > > > > > > > > > > > > > > > > > > > > > > > Dan Bress > > > > > > > > > Software Engineer > > > > > > > > > ONYX Consulting Services > > > > > > > > >
