That would have been great - the only things I would add are reading/writing attributes & reading the processor's properties. Maybe something about testing in NiFi using GenerateFlowFile too?
I ended up referring to the bundled processors a lot, but it was sometimes hard to see the wood for the trees. On Fri, 25 Jan 2019 at 23:04, Andy LoPresto <alopre...@apache.org> wrote: > > James, > > I’m wondering if a page outlining a toy processor (something like `CountText` > or `ReverseContent`) and doing a line-by-line annotation from a developer’s > perspective would be helpful. It could be a few sections: > > 1. How to get to this point > * running the maven archetype > * choosing the directory to install to > * putting the class name in the manifest file > * etc. > 2. The code > * here’s the class > * here’s what extending AbstractProcessor gets you, etc. A lot of > this is currently in the Developer Guide, but in textbook mode > * here’s how to modify some code (i.e. write one line of Java that > switches it from straight content pass-through to reversing the text) > * here’s how to make a unit test (introduce the TestRunner framework, > etc.) > 3. Running, building, installing > * Run your unit test from the IDE/maven > * Build the NAR module > * Install the NAR in NiFi lib/ or custom/ > * Restart NiFi > * See the NAR loaded in the log > * Deploy the component on the canvas > > I imagine this being written more conversationally/blog-like than most of our > current reference documentation to be used as a split-screen walkthrough. > Each section could certainly link to the existing detailed documentation for > various topics, like the processor lifecycle, etc. > > Does this sounds like something that would have helped you? > > Andy LoPresto > alopre...@apache.org > alopresto.apa...@gmail.com > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > On Jan 25, 2019, at 1:59 PM, James Srinivasan <james.sriniva...@gmail.com> > > wrote: > > > > 9) Oh, and the wiki is a little hard to navigate and the contents rather > > patchy > > > > On Fri, 25 Jan 2019 at 21:57, James Srinivasan > > <james.sriniva...@gmail.com> wrote: > >> > >> As someone relatively new to NiFi dev, here's my £0.02. (Yes, I > >> realise I could and possibly should submit PRs :) > >> > >> 1) I'm used to Java and Maven, so used the archetype. It worked fine, > >> it would have been nice it if set up unit tests for me. > >> 2) The User and Developer documentation is great and comprehensive. > >> Finding the developer docs is a little painful (handful of items at > >> the end of a scrolling list of 200+ processors) > >> 3) The Developer docs could possibly do with a little more clarity on > >> processor lifetime i.e. what is called when ^h^h^h - skimming back > >> over the docs, it looks pretty clear now > >> 4) Some example code for common operations e.g. getting/setting > >> attributes or reading/writing/modifying flowfile content would be > >> great. > >> 5) When using existing processors for inspiration, best practices > >> weren't always clear e.g. some generated properties inside > >> getSupportedPropertyDescriptors(), others generated a private static > >> list on init and returned that. Such differences are inevitable in a > >> large project, but it would be nice to have something blessed to start > >> from. > >> 6) (Minor niggle - layout of the docs doesn't work great on a phone screen) > >> 7) I couldn't find (m?)any docs about the Groovy scripting API, but > >> the great blog posts by Matt Burgess and others were invaluable > >> 8) In case this all sounds too negative, NiFi is fab! > >> > >> On Fri, 25 Jan 2019 at 18:47, Andrew Grande <apere...@gmail.com> wrote: > >>> > >>> I am not against the archetype. But we need to spell out every step of the > >>> way. I'd like to see a user thinking about their custom logic ASAP rather > >>> than fighting the tools to get started. Those steps should be brain-dead, > >>> just reflexes, if you know what I mean. Hell, let them create a custom > >>> processor project or prototype in a script by accident even! :) > >>> > >>> On Fri, Jan 25, 2019, 10:43 AM Bryan Bende <bbe...@gmail.com> wrote: > >>> > >>>> That makes sense about the best practice for deploying to an > >>>> additional lib directory. > >>>> > >>>> So for the project structure you are saying it would be easier to have > >>>> a repo somewhere with essentially the same thing that is in the > >>>> archetype, but they just clone it and rename it themselves (what the > >>>> archetype does for you)? > >>>> > >>>> Something that I think would be awesome is if we could provide a > >>>> web-based project initializer that would essentially run the archetype > >>>> behind the scenes and then let you download the archive of the code, > >>>> just like the spring-boot starter [1]. Not sure if their initializr is > >>>> something that can be re-used and customized [2]. > >>>> > >>>> The problem is we would need to host that somewhere. > >>>> > >>>> [1] https://start.spring.io/ > >>>> [2] https://github.com/spring-io/initializr > >>>> > >>>> On Fri, Jan 25, 2019 at 12:56 PM Andrew Grande <apere...@gmail.com> > >>>> wrote: > >>>>> > >>>>> We assume they create new projects from archetypes every day. They > >>>>> don't. > >>>>> > >>>>> We also assume they know how to deploy new NARs. Most don't. Especially > >>>> if > >>>>> we want them to follow best practices and create an additional NAR > >>>> bundles > >>>>> directory entry im the config (vs dumping into nifi lib). > >>>>> > >>>>> I can attest that I feel a bit lost myself every time I need to come > >>>>> back > >>>>> to this and refresh my brain synapses. If we could make these not > >>>>> require > >>>>> any of that and make simple thinga dead simple.... > >>>>> > >>>>> Andrew > >>>>> > >>>>> On Fri, Jan 25, 2019, 9:47 AM Bryan Bende <bbe...@gmail.com> wrote: > >>>>> > >>>>>> Andrew, > >>>>>> > >>>>>> I'm not disagreeing with your points, but I'm curious how you see > >>>>>> those two ideas being different from the processor archetype and the > >>>>>> wiki page with the archetype commands? > >>>>>> > >>>>>> Is it just that people don't know about it? > >>>>>> > >>>>>> -Bryan > >>>>>> > >>>>>> [1] > >>>>>> > >>>> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions > >>>>>> > >>>>>> On Fri, Jan 25, 2019 at 12:23 PM Otto Fowler <ottobackwa...@gmail.com> > >>>>>> wrote: > >>>>>>> > >>>>>>> I think this ties into my other discuss thread on refreshing the > >>>>>> archetypes > >>>>>>> > >>>>>>> > >>>>>>> On January 25, 2019 at 11:50:10, Andrew Grande (apere...@gmail.com) > >>>>>> wrote: > >>>>>>> > >>>>>>> I consistently see my users struggling when they move up the nifi > >>>> food > >>>>>>> chain and start looking at custom processors. The good content about > >>>>>>> prototyping processsors via scripting processors and finalizing with > >>>> a > >>>>>> full > >>>>>>> NAR bundle is everywhere but where it should be. > >>>>>>> > >>>>>>> A few simple changes could help (not *more* docs). They are great, > >>>> much > >>>>>>> better than in many other projecta, but people are already drowning > >>>> in > >>>>>>> those. > >>>>>>> > >>>>>>> How about: > >>>>>>> > >>>>>>> + ISP has a pre-populated processor sceleton. A simple no-op to fill > >>>> in > >>>>>> is > >>>>>>> miles better than a blank text area (which invokes a blank stare). > >>>>>>> > >>>>>>> + As much as we may loook down on this, but... A simple guide to a > >>>> full > >>>>>> NAR > >>>>>>> build as a series of copy/paste commands. > >>>>>>> > >>>>>>> There's more, but this should fit the context for now. > >>>>>>> > >>>>>>> Andrew > >>>>>>> > >>>>>>> On Fri, Jan 25, 2019, 8:13 AM Mike Thomsen <mikerthom...@gmail.com> > >>>>>> wrote: > >>>>>>> > >>>>>>>> One of the changes we should make is to create a separate guide for > >>>>>>> product > >>>>>>>> vendors on how to build and maintain a bundle. We're at that point > >>>>>> where > >>>>>>>> vendors will have to do it on their own as extension providers, so > >>>> it > >>>>>>> would > >>>>>>>> be very helpful for them to have a simple and straight forward > >>>> document > >>>>>>>> showing them what should be there, best practices for > >>>> maintainability > >>>>>> and > >>>>>>>> where to announce it. > >>>>>>>> > >>>>>>>> On Fri, Jan 25, 2019 at 9:59 AM Bryan Bende <bbe...@gmail.com> > >>>> wrote: > >>>>>>>> > >>>>>>>>> I think we have a lot more documentation than most projects, but > >>>> I > >>>>>>>>> think an issue is that content is scattered in many different > >>>>>>>>> locations, and some of the docs are huge reference guides where > >>>> it > >>>>>> can > >>>>>>>>> be hard to find all the pieces of what you are trying to do. > >>>>>>>>> > >>>>>>>>> The first thing a new contributor wants to do is get the code > >>>> and run > >>>>>>>>> a build, and we do have a quick-start guide linked to on the > >>>> site, > >>>>>> but > >>>>>>>>> I think there is a lot of extra information in there that is not > >>>>>>>>> really relevant to someone just wanting get the code and build. > >>>> We > >>>>>>>>> could have separate guides per OS like "Build NiFi on Linux", > >>>> "Build > >>>>>>>>> NiFi on Windows", etc, where each guide was 4-5 steps like: > >>>>>>>>> > >>>>>>>>> - Clone repo > >>>>>>>>> - checkout master > >>>>>>>>> - run maven > >>>>>>>>> - cd to assembly > >>>>>>>>> - ./bin/nifi.sh > >>>>>>>>> > >>>>>>>>> The next thing they want to do is contribute a change, and we > >>>> have a > >>>>>>>>> great contributor guide, but again I think there could be a very > >>>>>> short > >>>>>>>>> tutorial for the most common steps: > >>>>>>>>> > >>>>>>>>> - fork repo > >>>>>>>>> - clone fork > >>>>>>>>> - create branch > >>>>>>>>> - make changes > >>>>>>>>> - push branch > >>>>>>>>> - submit pr > >>>>>>>>> > >>>>>>>>> and then say something like "for a more detailed description of > >>>> the > >>>>>>>>> contribution process, please reference the Contributor Guide". > >>>>>>>>> > >>>>>>>>> If we then make these getting started guides more prominent > >>>> right in > >>>>>>>>> the middle of the NiFi homepage, then maybe they will be easier > >>>> to > >>>>>>>>> find for new community members. > >>>>>>>>> > >>>>>>>>> We can keep extending this idea to other common tasks beyond just > >>>>>>>>> building and contributing. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Thu, Jan 24, 2019 at 8:03 PM Andy LoPresto < > >>>> alopre...@apache.org> > >>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>> Hi folks, > >>>>>>>>>> > >>>>>>>>>> Based on some recent (and long-term) experiences, I wanted to > >>>>>> discuss > >>>>>>>>> with the community what we could do to lower the barrier of > >>>> entry to > >>>>>>>> using > >>>>>>>>> & contributing to NiFi. I hope to get some good feedback from > >>>> both > >>>>>>>>> long-time and newer members, and determine some immediate > >>>> concrete > >>>>>>> steps > >>>>>>>> we > >>>>>>>>> can take. > >>>>>>>>>> > >>>>>>>>>> Problems identified: > >>>>>>>>>> * NiFi has a number of custom profiles, so a simple “mvn clean > >>>>>>> install” > >>>>>>>>> in project root doesn’t get a new developer up and running > >>>>>> immediately > >>>>>>>>>> * The API is very well defined, but for new contributors, it > >>>> can > >>>>>> be a > >>>>>>>>> challenge to know where to put functionality, and building a > >>>> custom > >>>>>>>>> processor + NAR and deploying isn’t a one-step process > >>>>>>>>>> * Project size (and build size/time) is large. This can > >>>> restrict > >>>>>> the > >>>>>>>>> minimum hardware necessary, elongate the development cycle, etc. > >>>>>>>>>> * Some new users do not receive mailing list replies > >>>>>>>>>> > >>>>>>>>>> Possible solutions: > >>>>>>>>>> * On a clean git clone, “mvn clean install” should build a > >>>> working > >>>>>>>>> instance. Maybe we provide a quickstart.sh script to handle the > >>>>>> default > >>>>>>>>> maven build, change to the target directory, and start NiFi? > >>>>>>>>>> * Individual contributors have written excellent blogs, and > >>>>>>>>> documentation exists, but making it more prominent or more easily > >>>>>>>> accessed > >>>>>>>>> could help? > >>>>>>>>>> * Extension registry will solve all the world’s problems > >>>> (related > >>>>>> to > >>>>>>>>> bundling and build time) > >>>>>>>>>> * Not sure about this one — I don’t know if it’s because > >>>> they’re > >>>>>> not > >>>>>>>>> subscribed, their mail client is blocking them, etc. > >>>>>>>>>> > >>>>>>>>>> I’ve said my bit, now I am eager to hear from other community > >>>>>> members > >>>>>>>> on > >>>>>>>>> their experiences, steps that helped them, and suggestions for > >>>> the > >>>>>>> future > >>>>>>>>> to continue to make the NiFi community welcoming to new users. > >>>>>> Thanks. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Andy LoPresto > >>>>>>>>>> alopre...@apache.org > >>>>>>>>>> alopresto.apa...@gmail.com > >>>>>>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D > >>>> EF69 > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>> > >>>> >
