Drew, The vast majority of the documentation for Processors already exists within the NiFi API. For example, each Property of a Processor is defined with a PropertyDescription that contians a a name, description, whether or not the Expression Language is supported, etc. We did this via the API specifically because we want to use this information from within the User Interface to provide specific information when processors are being configured. Not sure that JavaDoc doclets can meet that need... then again, I don't know a whole lot about doclets. If you think they may provide this sort of functionality, I could look more into it. Thanks-Mark
> From: [email protected] > Date: Mon, 19 Jan 2015 12:54:43 -0500 > Subject: Re: Processor documentation plugin > To: [email protected] > > Hi All, > > Getting in on this thread a bit late. This is interesting work. > > Is this functionality still slated to be rolled into a maven plugin? Could > this be something that's implemented as a javadoc doclet, or does that api > tie everything too closely to the static approach? > > Drew > > On Mon, Jan 19, 2015 at 11:49 AM, Daniel Bress <[email protected]> > wrote: > > > Mark, > > I've got the API for generating > > Processor/ControllerService/ReportingTask documentation, and now am looking > > at integrating that into the Framework during startup. I've got a few > > questions about how to tie everything together. > > > > 1) Where should my code reside in the tree? I was going to create a new > > jar/project under framework-bundle/framework called documentation. > > 2) My stuff requires initialized instances of ConfigurableComponents. The > > reason being most Processors set up their properties in their init() > > methods. So what's the best way to get an instance of a > > Processor/ControllerService/Reporting task? I was planning on kicking my > > stuff out of the nifi-runtime/NiFi class. To get things going I've been > > using stubbed out versions of the InitializationContext interfaces that I > > wrote. Should I get the instances from the FlowController? > > > > I've got it tied in right now in a maybe hacky way, but the docs looks > > good. I went through all the existing html, and deleted anything that > > didn't add additional detail. Anything that did provide additional detail > > I renamed to "additionalDetails.html" which my code detects and links from > > the generated documentation. Also having NIFI-6 and NIFI-4 resolved will > > make the documentation look better for ControllerServices and > > ReportingTasks. > > > > Once I get it integrated and tested a bit I'll add the patch to the > > ticket. Thanks for your help! > > > > Dan Bress > > Software Engineer > > ONYX Consulting Services > > > > ________________________________________ > > From: Daniel Bress <[email protected]> > > Sent: Friday, January 16, 2015 4:14 PM > > To: [email protected] > > Subject: Re: Processor documentation plugin > > > > Mark & Matt, > > Cool. While I like the idea of @SupportsDynamicProperties, especially > > if it has a description, I'd also like a guaranteed way to get the dynamic > > PropertyDescriptor so I can interrogate it further. I'd be interested in > > knowing whether it supports expression language, and/or how the value of > > the property is validated. > > > > On that note, I was thinking it may be helpful to add either an annotation > > or a method to the Validator interface that could provide a text > > description about what it is validating. Is this useful or overkill? > > > > Dan Bress > > Software Engineer > > ONYX Consulting Services > > > > ________________________________________ > > From: Mark Payne <[email protected]> > > Sent: Friday, January 16, 2015 11:14 AM > > To: [email protected] > > Subject: RE: Processor documentation plugin > > > > It might make sense to have an annotation for this: > > @SupportsDynamicProperties > > Perhaps even allow developer to provide a string for how they are used: > > @SupportsDynamicProperties("Name of dynamic property becomes attribute > > name; value becomes attribute value.") > > I've begun ticket #6: https://issues.apache.org/jira/browse/NIFI-6 but I > > have not pushed anything to a branch or anything yet. Basically, I > > deprecated the existing annotations and created 3 new packages for them. > > Instead of org.apache.nifi.processor.annotation, we will have: > > > > org.apache.nifi.annotation.documentationorg.apache.nifi.annotation.behaviororg.apache.nifi.annotation.lifecycle > > So the @SupportsDynamicProperties could go under 'behavior' if that makes > > sense. > > > > > Date: Fri, 16 Jan 2015 11:09:28 -0500 > > > Subject: Re: Processor documentation plugin > > > From: [email protected] > > > To: [email protected] > > > > > > Dan, > > > > > > We discussed item 1 awhile back and even created a ticket for it. > > However, > > > there was always higher priority work that took precedence. I am not sure > > > if that ticket was ever created in JIRA for Apache NiFi. If not, we > > should. > > > I believe the current mechanism allows the processor to decide if a given > > > dynamic property should be a supported. I think we would need a more > > > general indicator that dynamic properties are supported at all to trigger > > > that button state. This would likely help with item 2 as well. > > > > > > Matt > > > > > > On Fri, Jan 16, 2015 at 10:45 AM, Daniel Bress <[email protected]> > > > wrote: > > > > > > > Mark, > > > > Sounds good. I'm on board. I was thinking of having the > > > > ControllerServices/ReportingTask's documentation include an XML > > template, > > > > but if those will be configurable via the UI then that will not be > > needed. > > > > I'm on board with your rev 1 / rev 2 suggestions too. I'll get to > > work > > > > accordingly. > > > > > > > > > > > > Matt/Mark, > > > > Two UI/API questions: > > > > 1) If a processor does not support dynamic properties, why does the UI > > let > > > > them hit [+] New Property? > > > > 2) If I have an instance of Processor class, I can call > > > > getSupportedProperties() on it to get a list of the required/optional > > > > properties. But I don't have a way to know if that processor supports > > > > dynamic properties or not. Supporting dynamic properties or not was > > > > something I was planning on reflecting in the documentation. Is there > > any > > > > way to detect this currently? > > > > > > > > Thanks, > > > > Dan Bress > > > > Software Engineer > > > > ONYX Consulting Services > > > > > > > > ________________________________________ > > > > From: Mark Payne <[email protected]> > > > > Sent: Wednesday, January 14, 2015 8:51 PM > > > > To: [email protected] > > > > Subject: RE: Processor documentation plugin > > > > > > > > Dan, > > > > I think this is great! > > > > I would recommend we slate this for the 0.1.0 version, which includes > > > > putting Controller Services and Reporting Tasks in the UI. But by the > > same > > > > token I think that makes your first bulletin for "rev 2" more of a > > priority > > > > for "rev 1" if we can get it there. I could pitch in and help if > > needed. > > > > I'd also recommend you put the Capability Description in for Rev 1 as > > > > that's arguably the most important part of the docs (which may make the > > > > Tags very easy at that point??) > > > > The rest of what you have listed for "rev 2" i would agree should be > > there. > > > > > > > > Very excited to see this happening! > > > > Thanks-Mark > > > > > > > > > From: [email protected] > > > > > To: [email protected] > > > > > Subject: Re: Processor documentation plugin > > > > > Date: Thu, 15 Jan 2015 01:39:44 +0000 > > > > > > > > > > Mark et al, > > > > > Ok, I think I hear everyone's concerns and ideas. I will > > challenge > > > > myself to come up with a solution that makes everyone happy all the > > time. > > > > > > > > > > The goals of this effort will be: > > > > > - Machine generate as much of the basic(processor description, > > > > property description, relationship description) info as possible. This > > > > will make the developers lives better by storing this information in > > one > > > > place, and make users lives better by having the "inline" documentation > > > > match the "usage" documentation. This will also ensure that all the > > > > documentation is formatted consistently. > > > > > - Allow the developer to optionally provide additional html and > > > > images that would be linked from the machine generated documentation > > > > > - Provide a way for the documentation to be generated > > statically > > > > and linked from the Apache NiFi webpage > > > > > > > > > > A few other 'nice-to-have' goals(probably for rev 2): > > > > > - Support documenting ControllerServices and ReportingTasks is > > a > > > > similar way > > > > > -Maybe leverage existing annotations on these types? E.g > > > > @Tags, @CapabilityDescription... > > > > > - Provide new 'documentation' annotations that would allow a > > > > developer to describe the FlowFile attributes a processor uses on > > input, > > > > and which attributes a processor modifies on output > > > > > - Provide new 'documentation' annotations that describe the > > > > expected input format of the flow file content > > > > > - Provide new 'documentation' annotations that describe whether > > > > the processor modifies the flow file content > > > > > - Provide new 'documentation' annotations that describe the > > format > > > > of the outbound flow file content > > > > > - Provide new 'documentation' annotations that describe other > > NiFi > > > > artifacts(processors, controller services, reporting tasks) that are > > > > related to this one, e.g. "See Also ..." > > > > > - Consider if/how current Processor annotations should be > > > > displayed (@SideEffectFree, @TriggerSerially, @TriggerWhenEmpty..., > > > > @EventDriven) > > > > > > > > > > > > > > > High level implementation plan: > > > > > - Develop a utility class to read an AbstractProcessor(and > > > > existing HTML documention) and generated the basic documentation with > > links > > > > to existing HTML > > > > > - Integrate this class into the framework such that all > > > > documentation displayed within a NiFi instance is generated by the > > utility > > > > > - Integrate this utility into something that can take a NiFi > > > > build, dig through the nars, find the > > > > Processors/ControllerServices/ReportingTasks and generate their > > appropriate > > > > HTML documentation so that it can be stored somewhere and linked from > > the > > > > Apache NiFi website. > > > > > > > > > > If this feels like the wrong path, or if you have any other ideas > > > > please speak up. Thanks! > > > > > > > > > > Dan Bress > > > > > Software Engineer > > > > > ONYX Consulting Services > > > > > > > > > > ________________________________________ > > > > > From: Mark Payne <[email protected]> > > > > > Sent: Monday, January 12, 2015 3:40 PM > > > > > To: [email protected] > > > > > Subject: RE: Processor documentation plugin > > > > > > > > > > Dan, > > > > > Yes, that's what I'm saying. Right now, it's all done manually by > > hand, > > > > and it's a pain. The down side i think to doing it via a Maven plugin > > is > > > > that if the user wants to modify it, they have to rebuild, pull the > > > > generated html out of the target directory, and then modify it and put > > it > > > > in the src/main/resources/docs. Then, if they change something they > > have to > > > > do this again. If it's done on-the-fly, rather than building static > > content > > > > with a plugin, that doesn't have to happen. > > > > > But you may come up with something much better than I'm envisioning > > :) > > > > Having a util to do it is a good call though. Then it could be used in > > a > > > > maven plugin, by the app, or wherever is most appropriate. > > > > > Thanks-Mark > > > > > > > > > > > From: [email protected] > > > > > > To: [email protected] > > > > > > Subject: Re: Processor documentation plugin > > > > > > Date: Mon, 12 Jan 2015 20:34:03 +0000 > > > > > > > > > > > > Mark, > > > > > > Sorry.. I missed your reply in the thread. > > > > > > > > > > > > Are you saying that going forward, you want to auto generate the > > > > doc and allow the developer to include a "more info" link? I wanted to > > > > confirm that it doesn't work that way now. > > > > > > > > > > > > I'll think about this a little more. Might write some code to > > > > generate the documentation from an AbstractProcessor class, then decide > > > > later where to integrate it(maven plugin, or somewhere else) > > > > > > > > > > > > Thanks! > > > > > > > > > > > > Dan Bress > > > > > > Software Engineer > > > > > > ONYX Consulting Services > > > > > > > > > > > > ________________________________________ > > > > > > From: Mark Payne <[email protected]> > > > > > > Sent: Sunday, January 11, 2015 1:23 PM > > > > > > To: [email protected] > > > > > > Subject: Re: Processor documentation plugin > > > > > > > > > > > > Dan, > > > > > > > > > > > > Matt and I discussed this approach before but ultimately decided > > that > > > > we like the idea of just auto generating the docs on demand with a > > "More > > > > Info" type of link that lets you add more in html format if you care > > to do > > > > so. > > > > > > > > > > > > Others' thoughts and ideas are welcome though! > > > > > > > > > > > > Sent from my iPhone > > > > > > > > > > > > > On Jan 11, 2015, at 12:32 PM, Daniel Bress < > > [email protected]> > > > > wrote: > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > Just pulled down the code out of Git and got everything up and > > > > running on Windows. This was pretty simple, and the UI looks great > > under > > > > chrome. > > > > > > > > > > > > > > > > > > > > > I was thinking of writing a maven plugin to generate the > > processor > > > > usage HTML documentation. > > > > > > > > > > > > > > > > > > > > > Looking through the existing processor documentation, it seems > > > > like most of this information comes directly from the code in the > > > > processors, via @CapabilityDescription, and the PropertyDescriptors and > > > > Relationships. Then if your processor only requires this level of > > > > documentation in the HTML format, you could tie this plugin to the > > build, > > > > and keep your inline processor documentation in sync with the HTML > > > > documentation. > > > > > > > > > > > > > > > > > > > > > What do you think? > > > > > > > > > > > > > > > > > > > > > Dan Bress > > > > > > > Software Engineer > > > > > > > ONYX Consulting Services > > > > > >
