Dan, For whatever reason, when I follow the hyperlinks, the last "slash" is a \ instead of a / and I get a 404 ( http://danbress.github.io/generated-documentation/components/org.apache.nifi.processors.standard.InvokeHTTP%5Cindex.html).
On Sat, Jan 24, 2015 at 1:01 PM, Daniel Bress <[email protected]> wrote: > All, > I've generated the first round of documentation I'd like everyone to > look at to see if they like this new format. Its really only slightly > changed from the current format(using tables instead of lists for > Properties and Relationships), but its all machine generated, except for > the stuff linked by "Additional Details". > > Again, this would be generated by the app at startup. I'd also like to > have this generated sometime during the build and linked from the website. > For now I just pulled that generated html and uploaded it to my github page > for your review. > > http://danbress.github.io/generated-documentation/components/ > > Let me know what you think! I'm open to any ideas/suggestions. > > Dan Bress > Software Engineer > ONYX Consulting Services > > ________________________________________ > From: Mark Payne <[email protected]> > Sent: Thursday, January 22, 2015 8:16 AM > To: [email protected] > Subject: RE: Processor documentation plugin > > Dan, > OK, now I understand where you're coming from. Sorry, I misunderstood the > idea before. If you think it would be too verbose then I think we should > hold off for now. I think it's best to keep it simple. If we want to add it > in later, we can. If we add it now and decide it's too verbose, it would be > awkward to go back and remove documentation later but it's easy to later > add in something that's missing. > > So in short I agree with your assessment :) > -Mark > > > From: [email protected] > > To: [email protected] > > Subject: Re: Processor documentation plugin > > Date: Thu, 22 Jan 2015 03:13:16 +0000 > > > > Mark, > > No worries. On the dynamic property tip, I see what you are saying. > I was hoping to provide the same level of detail in describing dynamic > properties as non-dynamic ones, but right now I see no way to do that. > Thats fine, I'll put that part(providing detailed documentation of the > dynamic properties) on the back burner. > > > > On the Validator tip, for each PropertyDescriptor I am getting the > list of AllowableValues so that I can say "This property has X, Y and Z for > valid values". If there are no AllowableValues present I would like to > look at the validators and say "This property requires a positive > integer". or "This requires a <data size> followed by a <data unit>". > Although typing this all out now makes it seem like it might be too verbose > and cluttered. I think I'll forget this idea and just work with > AllowableValues. > > > > Thanks again, > > > > Dan Bress > > Software Engineer > > ONYX Consulting Services > > > > ________________________________________ > > From: Mark Payne <[email protected]> > > Sent: Tuesday, January 20, 2015 12:16 PM > > To: [email protected] > > Subject: RE: Processor documentation plugin > > > > Dan, > > Sorry, I never did respond to this e-mail. > > You cannot get the dynamic PropertyDescriptor because the > AbstractConfigurableComponent (the base abstract class from which every > processor, controller service, reporting task that I know of extends) > provides the following method signature: > > protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final > String propertyDescriptorName) > > I.e., there is not a PropertyDescriptor for dynamic properties but > rather the descriptor can be different for every dynamic property. For > example, consider a Processor that allows dynamic properties but for every > property X there must be 2 other properties: X.format (any string) and > X.size (a non-zero integer). If there were a single PropertyDescriptor, > this wouldn't be possible because all dynamic properties would have the > same validators, etc. But by making a property descriptor based on the > property name, we can easily achieve this. > > Regarding the second note: I don't think Validator can provide > information about what it is validating because it doesn't know. For > instance, PositiveInteger validator just validates that some value is a > positive integer. It does not (and I think should not) know the context of > the value. > > If anything here doesn't make sense or if you think it could be done > better, let me know and we can discuss further. > > Thanks-Mark > > > > > From: [email protected] > > > To: [email protected] > > > Subject: Re: Processor documentation plugin > > > Date: Fri, 16 Jan 2015 21:14:41 +0000 > > > > > > Mark & Matt, > > > Cool. While I like the idea of @SupportsDynamicProperties, > especially if it has a description, I'd also like a guaranteed way to get > the dynamic PropertyDescriptor so I can interrogate it further. I'd be > interested in knowing whether it supports expression language, and/or how > the value of the property is validated. > > > > > > On that note, I was thinking it may be helpful to add either an > annotation or a method to the Validator interface that could provide a text > description about what it is validating. Is this useful or overkill? > > > > > > Dan Bress > > > Software Engineer > > > ONYX Consulting Services > > > > > > ________________________________________ > > > From: Mark Payne <[email protected]> > > > Sent: Friday, January 16, 2015 11:14 AM > > > To: [email protected] > > > Subject: RE: Processor documentation plugin > > > > > > It might make sense to have an annotation for this: > > > @SupportsDynamicProperties > > > Perhaps even allow developer to provide a string for how they are used: > > > @SupportsDynamicProperties("Name of dynamic property becomes attribute > name; value becomes attribute value.") > > > I've begun ticket #6: https://issues.apache.org/jira/browse/NIFI-6 > but I have not pushed anything to a branch or anything yet. Basically, I > deprecated the existing annotations and created 3 new packages for them. > Instead of org.apache.nifi.processor.annotation, we will have: > > > > org.apache.nifi.annotation.documentationorg.apache.nifi.annotation.behaviororg.apache.nifi.annotation.lifecycle > > > So the @SupportsDynamicProperties could go under 'behavior' if that > makes sense. > > > > > > > Date: Fri, 16 Jan 2015 11:09:28 -0500 > > > > Subject: Re: Processor documentation plugin > > > > From: [email protected] > > > > To: [email protected] > > > > > > > > Dan, > > > > > > > > We discussed item 1 awhile back and even created a ticket for it. > However, > > > > there was always higher priority work that took precedence. I am not > sure > > > > if that ticket was ever created in JIRA for Apache NiFi. If not, we > should. > > > > I believe the current mechanism allows the processor to decide if a > given > > > > dynamic property should be a supported. I think we would need a more > > > > general indicator that dynamic properties are supported at all to > trigger > > > > that button state. This would likely help with item 2 as well. > > > > > > > > Matt > > > > > > > > On Fri, Jan 16, 2015 at 10:45 AM, Daniel Bress < > [email protected]> > > > > wrote: > > > > > > > > > Mark, > > > > > Sounds good. I'm on board. I was thinking of having the > > > > > ControllerServices/ReportingTask's documentation include an XML > template, > > > > > but if those will be configurable via the UI then that will not be > needed. > > > > > I'm on board with your rev 1 / rev 2 suggestions too. I'll get > to work > > > > > accordingly. > > > > > > > > > > > > > > > Matt/Mark, > > > > > Two UI/API questions: > > > > > 1) If a processor does not support dynamic properties, why does > the UI let > > > > > them hit [+] New Property? > > > > > 2) If I have an instance of Processor class, I can call > > > > > getSupportedProperties() on it to get a list of the > required/optional > > > > > properties. But I don't have a way to know if that processor > supports > > > > > dynamic properties or not. Supporting dynamic properties or not > was > > > > > something I was planning on reflecting in the documentation. Is > there any > > > > > way to detect this currently? > > > > > > > > > > Thanks, > > > > > Dan Bress > > > > > Software Engineer > > > > > ONYX Consulting Services > > > > > > > > > > ________________________________________ > > > > > From: Mark Payne <[email protected]> > > > > > Sent: Wednesday, January 14, 2015 8:51 PM > > > > > To: [email protected] > > > > > Subject: RE: Processor documentation plugin > > > > > > > > > > Dan, > > > > > I think this is great! > > > > > I would recommend we slate this for the 0.1.0 version, which > includes > > > > > putting Controller Services and Reporting Tasks in the UI. But by > the same > > > > > token I think that makes your first bulletin for "rev 2" more of a > priority > > > > > for "rev 1" if we can get it there. I could pitch in and help if > needed. > > > > > I'd also recommend you put the Capability Description in for Rev 1 > as > > > > > that's arguably the most important part of the docs (which may > make the > > > > > Tags very easy at that point??) > > > > > The rest of what you have listed for "rev 2" i would agree should > be there. > > > > > > > > > > Very excited to see this happening! > > > > > Thanks-Mark > > > > > > > > > > > From: [email protected] > > > > > > To: [email protected] > > > > > > Subject: Re: Processor documentation plugin > > > > > > Date: Thu, 15 Jan 2015 01:39:44 +0000 > > > > > > > > > > > > Mark et al, > > > > > > Ok, I think I hear everyone's concerns and ideas. I will > challenge > > > > > myself to come up with a solution that makes everyone happy all > the time. > > > > > > > > > > > > The goals of this effort will be: > > > > > > - Machine generate as much of the basic(processor > description, > > > > > property description, relationship description) info as possible. > This > > > > > will make the developers lives better by storing this information > in one > > > > > place, and make users lives better by having the "inline" > documentation > > > > > match the "usage" documentation. This will also ensure that all > the > > > > > documentation is formatted consistently. > > > > > > - Allow the developer to optionally provide additional > html and > > > > > images that would be linked from the machine generated > documentation > > > > > > - Provide a way for the documentation to be generated > statically > > > > > and linked from the Apache NiFi webpage > > > > > > > > > > > > A few other 'nice-to-have' goals(probably for rev 2): > > > > > > - Support documenting ControllerServices and > ReportingTasks is a > > > > > similar way > > > > > > -Maybe leverage existing annotations on these types? > E.g > > > > > @Tags, @CapabilityDescription... > > > > > > - Provide new 'documentation' annotations that would allow > a > > > > > developer to describe the FlowFile attributes a processor uses on > input, > > > > > and which attributes a processor modifies on output > > > > > > - Provide new 'documentation' annotations that describe the > > > > > expected input format of the flow file content > > > > > > - Provide new 'documentation' annotations that describe > whether > > > > > the processor modifies the flow file content > > > > > > - Provide new 'documentation' annotations that describe > the format > > > > > of the outbound flow file content > > > > > > - Provide new 'documentation' annotations that describe > other NiFi > > > > > artifacts(processors, controller services, reporting tasks) that > are > > > > > related to this one, e.g. "See Also ..." > > > > > > - Consider if/how current Processor annotations should be > > > > > displayed (@SideEffectFree, @TriggerSerially, @TriggerWhenEmpty..., > > > > > @EventDriven) > > > > > > > > > > > > > > > > > > High level implementation plan: > > > > > > - Develop a utility class to read an AbstractProcessor(and > > > > > existing HTML documention) and generated the basic documentation > with links > > > > > to existing HTML > > > > > > - Integrate this class into the framework such that all > > > > > documentation displayed within a NiFi instance is generated by the > utility > > > > > > - Integrate this utility into something that can take a > NiFi > > > > > build, dig through the nars, find the > > > > > Processors/ControllerServices/ReportingTasks and generate their > appropriate > > > > > HTML documentation so that it can be stored somewhere and linked > from the > > > > > Apache NiFi website. > > > > > > > > > > > > If this feels like the wrong path, or if you have any other > ideas > > > > > please speak up. Thanks! > > > > > > > > > > > > Dan Bress > > > > > > Software Engineer > > > > > > ONYX Consulting Services > > > > > > > > > > > > ________________________________________ > > > > > > From: Mark Payne <[email protected]> > > > > > > Sent: Monday, January 12, 2015 3:40 PM > > > > > > To: [email protected] > > > > > > Subject: RE: Processor documentation plugin > > > > > > > > > > > > Dan, > > > > > > Yes, that's what I'm saying. Right now, it's all done manually > by hand, > > > > > and it's a pain. The down side i think to doing it via a Maven > plugin is > > > > > that if the user wants to modify it, they have to rebuild, pull the > > > > > generated html out of the target directory, and then modify it and > put it > > > > > in the src/main/resources/docs. Then, if they change something > they have to > > > > > do this again. If it's done on-the-fly, rather than building > static content > > > > > with a plugin, that doesn't have to happen. > > > > > > But you may come up with something much better than I'm > envisioning :) > > > > > Having a util to do it is a good call though. Then it could be > used in a > > > > > maven plugin, by the app, or wherever is most appropriate. > > > > > > Thanks-Mark > > > > > > > > > > > > > From: [email protected] > > > > > > > To: [email protected] > > > > > > > Subject: Re: Processor documentation plugin > > > > > > > Date: Mon, 12 Jan 2015 20:34:03 +0000 > > > > > > > > > > > > > > Mark, > > > > > > > Sorry.. I missed your reply in the thread. > > > > > > > > > > > > > > Are you saying that going forward, you want to auto > generate the > > > > > doc and allow the developer to include a "more info" link? I > wanted to > > > > > confirm that it doesn't work that way now. > > > > > > > > > > > > > > I'll think about this a little more. Might write some code > to > > > > > generate the documentation from an AbstractProcessor class, then > decide > > > > > later where to integrate it(maven plugin, or somewhere else) > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > Dan Bress > > > > > > > Software Engineer > > > > > > > ONYX Consulting Services > > > > > > > > > > > > > > ________________________________________ > > > > > > > From: Mark Payne <[email protected]> > > > > > > > Sent: Sunday, January 11, 2015 1:23 PM > > > > > > > To: [email protected] > > > > > > > Subject: Re: Processor documentation plugin > > > > > > > > > > > > > > Dan, > > > > > > > > > > > > > > Matt and I discussed this approach before but ultimately > decided that > > > > > we like the idea of just auto generating the docs on demand with a > "More > > > > > Info" type of link that lets you add more in html format if you > care to do > > > > > so. > > > > > > > > > > > > > > Others' thoughts and ideas are welcome though! > > > > > > > > > > > > > > Sent from my iPhone > > > > > > > > > > > > > > > On Jan 11, 2015, at 12:32 PM, Daniel Bress < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > Just pulled down the code out of Git and got everything up > and > > > > > running on Windows. This was pretty simple, and the UI looks > great under > > > > > chrome. > > > > > > > > > > > > > > > > > > > > > > > > I was thinking of writing a maven plugin to generate the > processor > > > > > usage HTML documentation. > > > > > > > > > > > > > > > > > > > > > > > > Looking through the existing processor documentation, it > seems > > > > > like most of this information comes directly from the code in the > > > > > processors, via @CapabilityDescription, and the > PropertyDescriptors and > > > > > Relationships. Then if your processor only requires this level of > > > > > documentation in the HTML format, you could tie this plugin to the > build, > > > > > and keep your inline processor documentation in sync with the HTML > > > > > documentation. > > > > > > > > > > > > > > > > > > > > > > > > What do you think? > > > > > > > > > > > > > > > > > > > > > > > > Dan Bress > > > > > > > > Software Engineer > > > > > > > > ONYX Consulting Services > > > > > >
