Throw away :-) In our org, we have some people who can use NiFi to create flows with the built-in processors. I guess HWX would all them "Data Flow Managers". They might know some scripting, so can write Jython scripts.
We have far fewer (approx 1) person (me) who can write custom processors in Java. Sometimes, I also create flows. I'd like to have lots more of both kinds of people. "User" & "Extending NiFi" actually maps to the internal training I run, so I'd be happy :-) James On Mon, 28 Jan 2019 at 18:01, Andrew Grande <apere...@gmail.com> wrote: > > Not to throw in a monkey wrench, but does it really make sense to split > User and Developer? In all years I've never seen a user who wasn't a > developer. > > Maybe call it a User and Extending NiFi sections? > > On Mon, Jan 28, 2019, 9:18 AM James Srinivasan <james.sriniva...@gmail.com> > wrote: > > > Rather than lumping all the documentation together in a single huge > > doc, I was thinking separate entries in the top bar of the NiFi site > > under "Documentation" for something like: > > > > General (Overview & Getting Started) > > User (User Guide, Expression Language Guide, Record Path Guide & > > detailed Processor Usage) > > Admin (Admin Guide) > > Developer (All the text currently under Developer) > > > > Breaking it out into multiple top-level headings will hopefully help > > people find what they need more quickly e.g. with my Developer hat on, > > I don't much care about the details of the FooBarProcessor, whereas > > with my User hat on, I really want to know about its parameters and > > what they mean. Likewise, a non-admin probably doesn't much care about > > certificates etc. > > > > Does this makes sense? What do others think? > > > > On Mon, 28 Jan 2019 at 17:04, Bryan Bende <bbe...@gmail.com> wrote: > > > > > > Currently it’s broken into General and Developer, so were you thinking of > > > splitting General into User and Admin? > > > > > > On Mon, Jan 28, 2019 at 11:34 AM James Srinivasan < > > > james.sriniva...@gmail.com> wrote: > > > > > > > How about separating out User/Developer/Admin into separate docs? > > > > > > > > On Mon, 28 Jan 2019 at 16:13, Bryan Bende <bbe...@gmail.com> wrote: > > > > > > > > > > What does everyone think about bumping the "Developer" section of the > > > > > docs ahead of "Processors" so that it's easier to find? > > > > > > > > > > Here is what it would look like - > > > > > https://gist.github.com/bbende/279c983f5c80d4fad1431ae81862060f > > > > > > > > > > I also added links to the "Contributor Guide" and the "Maven > > Projects" > > > > > page since I think it would be helpful to make the Developer section > > > > > be the one-stop place people look for development help, although I > > can > > > > > see an argument for not mixing wiki content with the docs content. > > > > > > > > > > One issue would be if we want the docs to be fully usable in an > > > > > off-line environment, then linking to the wiki won't work, so we > > could > > > > > remove those links, or convert those pages to be part of the docs now > > > > > that they are more stable. > > > > > > > > > > For reference, we already have some links in the docs to the wiki: > > > > > > > > > > > > > > > > https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#supplying-a-contribution > > > > > > > > > > On Sat, Jan 26, 2019 at 10:49 AM Joe Witt <joe.w...@gmail.com> > > wrote: > > > > > > > > > > > > ...we can. but the discussion is about how to both lower the bar > > and > > > > offer > > > > > > more routes to the bar. > > > > > > > > > > > > On Sat, Jan 26, 2019, 10:45 AM Otto Fowler < > > ottobackwa...@gmail.com > > > > wrote: > > > > > > > > > > > > > Why wouldn’t we make the archetypes do this? > > > > > > > > > > > > > > > > > > > > > On January 25, 2019 at 18:04:25, Andy LoPresto ( > > alopre...@apache.org > > > > ) > > > > > > > wrote: > > > > > > > > > > > > > > James, > > > > > > > > > > > > > > I’m wondering if a page outlining a toy processor (something like > > > > > > > `CountText` or `ReverseContent`) and doing a line-by-line > > annotation > > > > from a > > > > > > > developer’s perspective would be helpful. It could be a few > > sections: > > > > > > > > > > > > > > 1. How to get to this point > > > > > > > * running the maven archetype > > > > > > > * choosing the directory to install to > > > > > > > * putting the class name in the manifest file > > > > > > > * etc. > > > > > > > 2. The code > > > > > > > * here’s the class > > > > > > > * here’s what extending AbstractProcessor gets you, etc. A lot of > > > > this is > > > > > > > currently in the Developer Guide, but in textbook mode > > > > > > > * here’s how to modify some code (i.e. write one line of Java > > that > > > > switches > > > > > > > it from straight content pass-through to reversing the text) > > > > > > > * here’s how to make a unit test (introduce the TestRunner > > > > framework, etc.) > > > > > > > 3. Running, building, installing > > > > > > > * Run your unit test from the IDE/maven > > > > > > > * Build the NAR module > > > > > > > * Install the NAR in NiFi lib/ or custom/ > > > > > > > * Restart NiFi > > > > > > > * See the NAR loaded in the log > > > > > > > * Deploy the component on the canvas > > > > > > > > > > > > > > I imagine this being written more conversationally/blog-like than > > > > most of > > > > > > > our current reference documentation to be used as a split-screen > > > > > > > walkthrough. Each section could certainly link to the existing > > > > detailed > > > > > > > documentation for various topics, like the processor lifecycle, > > etc. > > > > > > > > > > > > > > Does this sounds like something that would have helped you? > > > > > > > > > > > > > > Andy LoPresto > > > > > > > alopre...@apache.org > > > > > > > alopresto.apa...@gmail.com > > > > > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D > > EF69 > > > > > > > > > > > > > > > On Jan 25, 2019, at 1:59 PM, James Srinivasan < > > > > > > > james.sriniva...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > > 9) Oh, and the wiki is a little hard to navigate and the > > contents > > > > rather > > > > > > > patchy > > > > > > > > > > > > > > > > On Fri, 25 Jan 2019 at 21:57, James Srinivasan > > > > > > > > <james.sriniva...@gmail.com> wrote: > > > > > > > >> > > > > > > > >> As someone relatively new to NiFi dev, here's my £0.02. (Yes, > > I > > > > > > > >> realise I could and possibly should submit PRs :) > > > > > > > >> > > > > > > > >> 1) I'm used to Java and Maven, so used the archetype. It > > worked > > > > fine, > > > > > > > >> it would have been nice it if set up unit tests for me. > > > > > > > >> 2) The User and Developer documentation is great and > > > > comprehensive. > > > > > > > >> Finding the developer docs is a little painful (handful of > > items > > > > at > > > > > > > >> the end of a scrolling list of 200+ processors) > > > > > > > >> 3) The Developer docs could possibly do with a little more > > > > clarity on > > > > > > > >> processor lifetime i.e. what is called when ^h^h^h - skimming > > back > > > > > > > >> over the docs, it looks pretty clear now > > > > > > > >> 4) Some example code for common operations e.g. > > getting/setting > > > > > > > >> attributes or reading/writing/modifying flowfile content > > would be > > > > > > > >> great. > > > > > > > >> 5) When using existing processors for inspiration, best > > practices > > > > > > > >> weren't always clear e.g. some generated properties inside > > > > > > > >> getSupportedPropertyDescriptors(), others generated a private > > > > static > > > > > > > >> list on init and returned that. Such differences are > > inevitable > > > > in a > > > > > > > >> large project, but it would be nice to have something blessed > > to > > > > start > > > > > > > >> from. > > > > > > > >> 6) (Minor niggle - layout of the docs doesn't work great on a > > > > phone > > > > > > > screen) > > > > > > > >> 7) I couldn't find (m?)any docs about the Groovy scripting > > API, > > > > but > > > > > > > >> the great blog posts by Matt Burgess and others were > > invaluable > > > > > > > >> 8) In case this all sounds too negative, NiFi is fab! > > > > > > > >> > > > > > > > >> On Fri, 25 Jan 2019 at 18:47, Andrew Grande < > > apere...@gmail.com> > > > > wrote: > > > > > > > >>> > > > > > > > >>> I am not against the archetype. But we need to spell out > > every > > > > step of > > > > > > > the > > > > > > > >>> way. I'd like to see a user thinking about their custom logic > > > > ASAP > > > > > > > rather > > > > > > > >>> than fighting the tools to get started. Those steps should be > > > > > > > brain-dead, > > > > > > > >>> just reflexes, if you know what I mean. Hell, let them > > create a > > > > custom > > > > > > > >>> processor project or prototype in a script by accident even! > > :) > > > > > > > >>> > > > > > > > >>> On Fri, Jan 25, 2019, 10:43 AM Bryan Bende <bbe...@gmail.com > > > > > > > wrote: > > > > > > > >>> > > > > > > > >>>> That makes sense about the best practice for deploying to an > > > > > > > >>>> additional lib directory. > > > > > > > >>>> > > > > > > > >>>> So for the project structure you are saying it would be > > easier > > > > to have > > > > > > > >>>> a repo somewhere with essentially the same thing that is in > > the > > > > > > > >>>> archetype, but they just clone it and rename it themselves > > > > (what the > > > > > > > >>>> archetype does for you)? > > > > > > > >>>> > > > > > > > >>>> Something that I think would be awesome is if we could > > provide a > > > > > > > >>>> web-based project initializer that would essentially run the > > > > archetype > > > > > > > >>>> behind the scenes and then let you download the archive of > > the > > > > code, > > > > > > > >>>> just like the spring-boot starter [1]. Not sure if their > > > > initializr is > > > > > > > >>>> something that can be re-used and customized [2]. > > > > > > > >>>> > > > > > > > >>>> The problem is we would need to host that somewhere. > > > > > > > >>>> > > > > > > > >>>> [1] https://start.spring.io/ > > > > > > > >>>> [2] https://github.com/spring-io/initializr > > > > > > > >>>> > > > > > > > >>>> On Fri, Jan 25, 2019 at 12:56 PM Andrew Grande < > > > > apere...@gmail.com> > > > > > > > wrote: > > > > > > > >>>>> > > > > > > > >>>>> We assume they create new projects from archetypes every > > day. > > > > They > > > > > > > don't. > > > > > > > >>>>> > > > > > > > >>>>> We also assume they know how to deploy new NARs. Most > > don't. > > > > > > > Especially > > > > > > > >>>> if > > > > > > > >>>>> we want them to follow best practices and create an > > additional > > > > NAR > > > > > > > >>>> bundles > > > > > > > >>>>> directory entry im the config (vs dumping into nifi lib). > > > > > > > >>>>> > > > > > > > >>>>> I can attest that I feel a bit lost myself every time I > > need > > > > to come > > > > > > > back > > > > > > > >>>>> to this and refresh my brain synapses. If we could make > > these > > > > not > > > > > > > require > > > > > > > >>>>> any of that and make simple thinga dead simple.... > > > > > > > >>>>> > > > > > > > >>>>> Andrew > > > > > > > >>>>> > > > > > > > >>>>> On Fri, Jan 25, 2019, 9:47 AM Bryan Bende < > > bbe...@gmail.com> > > > > wrote: > > > > > > > >>>>> > > > > > > > >>>>>> Andrew, > > > > > > > >>>>>> > > > > > > > >>>>>> I'm not disagreeing with your points, but I'm curious how > > you > > > > see > > > > > > > >>>>>> those two ideas being different from the processor > > archetype > > > > and the > > > > > > > >>>>>> wiki page with the archetype commands? > > > > > > > >>>>>> > > > > > > > >>>>>> Is it just that people don't know about it? > > > > > > > >>>>>> > > > > > > > >>>>>> -Bryan > > > > > > > >>>>>> > > > > > > > >>>>>> [1] > > > > > > > >>>>>> > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions > > > > > > > >>>>>> > > > > > > > >>>>>> On Fri, Jan 25, 2019 at 12:23 PM Otto Fowler < > > > > > > > ottobackwa...@gmail.com> > > > > > > > > > > > > > > >>>>>> wrote: > > > > > > > >>>>>>> > > > > > > > >>>>>>> I think this ties into my other discuss thread on > > refreshing > > > > the > > > > > > > >>>>>> archetypes > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> On January 25, 2019 at 11:50:10, Andrew Grande ( > > > > apere...@gmail.com > > > > > > > ) > > > > > > > >>>>>> wrote: > > > > > > > >>>>>>> > > > > > > > >>>>>>> I consistently see my users struggling when they move up > > the > > > > nifi > > > > > > > >>>> food > > > > > > > >>>>>>> chain and start looking at custom processors. The good > > > > content > > > > > > > about > > > > > > > >>>>>>> prototyping processsors via scripting processors and > > > > finalizing > > > > > > > with > > > > > > > >>>> a > > > > > > > >>>>>> full > > > > > > > >>>>>>> NAR bundle is everywhere but where it should be. > > > > > > > >>>>>>> > > > > > > > >>>>>>> A few simple changes could help (not *more* docs). They > > are > > > > great, > > > > > > > >>>> much > > > > > > > >>>>>>> better than in many other projecta, but people are > > already > > > > drowning > > > > > > > >>>> in > > > > > > > >>>>>>> those. > > > > > > > >>>>>>> > > > > > > > >>>>>>> How about: > > > > > > > >>>>>>> > > > > > > > >>>>>>> + ISP has a pre-populated processor sceleton. A simple > > no-op > > > > to > > > > > > > fill > > > > > > > >>>> in > > > > > > > >>>>>> is > > > > > > > >>>>>>> miles better than a blank text area (which invokes a > > blank > > > > stare). > > > > > > > >>>>>>> > > > > > > > >>>>>>> + As much as we may loook down on this, but... A simple > > > > guide to a > > > > > > > >>>> full > > > > > > > >>>>>> NAR > > > > > > > >>>>>>> build as a series of copy/paste commands. > > > > > > > >>>>>>> > > > > > > > >>>>>>> There's more, but this should fit the context for now. > > > > > > > >>>>>>> > > > > > > > >>>>>>> Andrew > > > > > > > >>>>>>> > > > > > > > >>>>>>> On Fri, Jan 25, 2019, 8:13 AM Mike Thomsen < > > > > mikerthom...@gmail.com > > > > > > > > > > > > > > > >>>>>> wrote: > > > > > > > >>>>>>> > > > > > > > >>>>>>>> One of the changes we should make is to create a > > separate > > > > guide > > > > > > > for > > > > > > > >>>>>>> product > > > > > > > >>>>>>>> vendors on how to build and maintain a bundle. We're at > > > > that point > > > > > > > >>>>>> where > > > > > > > >>>>>>>> vendors will have to do it on their own as extension > > > > providers, so > > > > > > > >>>> it > > > > > > > >>>>>>> would > > > > > > > >>>>>>>> be very helpful for them to have a simple and straight > > > > forward > > > > > > > >>>> document > > > > > > > >>>>>>>> showing them what should be there, best practices for > > > > > > > >>>> maintainability > > > > > > > >>>>>> and > > > > > > > >>>>>>>> where to announce it. > > > > > > > >>>>>>>> > > > > > > > >>>>>>>> On Fri, Jan 25, 2019 at 9:59 AM Bryan Bende < > > > > bbe...@gmail.com> > > > > > > > >>>> wrote: > > > > > > > >>>>>>>> > > > > > > > >>>>>>>>> I think we have a lot more documentation than most > > > > projects, but > > > > > > > >>>> I > > > > > > > >>>>>>>>> think an issue is that content is scattered in many > > > > different > > > > > > > >>>>>>>>> locations, and some of the docs are huge reference > > guides > > > > where > > > > > > > >>>> it > > > > > > > >>>>>> can > > > > > > > >>>>>>>>> be hard to find all the pieces of what you are trying > > to > > > > do. > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> The first thing a new contributor wants to do is get > > the > > > > code > > > > > > > >>>> and run > > > > > > > >>>>>>>>> a build, and we do have a quick-start guide linked to > > on > > > > the > > > > > > > >>>> site, > > > > > > > >>>>>> but > > > > > > > >>>>>>>>> I think there is a lot of extra information in there > > that > > > > is not > > > > > > > >>>>>>>>> really relevant to someone just wanting get the code > > and > > > > build. > > > > > > > >>>> We > > > > > > > >>>>>>>>> could have separate guides per OS like "Build NiFi on > > > > Linux", > > > > > > > >>>> "Build > > > > > > > >>>>>>>>> NiFi on Windows", etc, where each guide was 4-5 steps > > like: > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> - Clone repo > > > > > > > >>>>>>>>> - checkout master > > > > > > > >>>>>>>>> - run maven > > > > > > > >>>>>>>>> - cd to assembly > > > > > > > >>>>>>>>> - ./bin/nifi.sh > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> The next thing they want to do is contribute a change, > > and > > > > we > > > > > > > >>>> have a > > > > > > > >>>>>>>>> great contributor guide, but again I think there could > > be > > > > a very > > > > > > > >>>>>> short > > > > > > > >>>>>>>>> tutorial for the most common steps: > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> - fork repo > > > > > > > >>>>>>>>> - clone fork > > > > > > > >>>>>>>>> - create branch > > > > > > > >>>>>>>>> - make changes > > > > > > > >>>>>>>>> - push branch > > > > > > > >>>>>>>>> - submit pr > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> and then say something like "for a more detailed > > > > description of > > > > > > > >>>> the > > > > > > > >>>>>>>>> contribution process, please reference the Contributor > > > > Guide". > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> If we then make these getting started guides more > > prominent > > > > > > > >>>> right in > > > > > > > >>>>>>>>> the middle of the NiFi homepage, then maybe they will > > be > > > > easier > > > > > > > >>>> to > > > > > > > >>>>>>>>> find for new community members. > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> We can keep extending this idea to other common tasks > > > > beyond just > > > > > > > >>>>>>>>> building and contributing. > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> On Thu, Jan 24, 2019 at 8:03 PM Andy LoPresto < > > > > > > > >>>> alopre...@apache.org> > > > > > > > >>>>>>>>> wrote: > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>>> Hi folks, > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>>> Based on some recent (and long-term) experiences, I > > > > wanted to > > > > > > > >>>>>> discuss > > > > > > > >>>>>>>>> with the community what we could do to lower the > > barrier of > > > > > > > >>>> entry to > > > > > > > >>>>>>>> using > > > > > > > >>>>>>>>> & contributing to NiFi. I hope to get some good > > feedback > > > > from > > > > > > > >>>> both > > > > > > > >>>>>>>>> long-time and newer members, and determine some > > immediate > > > > > > > >>>> concrete > > > > > > > >>>>>>> steps > > > > > > > >>>>>>>> we > > > > > > > >>>>>>>>> can take. > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>>> Problems identified: > > > > > > > >>>>>>>>>> * NiFi has a number of custom profiles, so a simple > > “mvn > > > > clean > > > > > > > >>>>>>> install” > > > > > > > >>>>>>>>> in project root doesn’t get a new developer up and > > running > > > > > > > >>>>>> immediately > > > > > > > >>>>>>>>>> * The API is very well defined, but for new > > contributors, > > > > it > > > > > > > >>>> can > > > > > > > >>>>>> be a > > > > > > > >>>>>>>>> challenge to know where to put functionality, and > > building > > > > a > > > > > > > >>>> custom > > > > > > > >>>>>>>>> processor + NAR and deploying isn’t a one-step process > > > > > > > >>>>>>>>>> * Project size (and build size/time) is large. This > > can > > > > > > > >>>> restrict > > > > > > > >>>>>> the > > > > > > > >>>>>>>>> minimum hardware necessary, elongate the development > > > > cycle, etc. > > > > > > > >>>>>>>>>> * Some new users do not receive mailing list replies > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>>> Possible solutions: > > > > > > > >>>>>>>>>> * On a clean git clone, “mvn clean install” should > > build a > > > > > > > >>>> working > > > > > > > >>>>>>>>> instance. Maybe we provide a quickstart.sh script to > > > > handle the > > > > > > > >>>>>> default > > > > > > > >>>>>>>>> maven build, change to the target directory, and start > > > > NiFi? > > > > > > > >>>>>>>>>> * Individual contributors have written excellent > > blogs, > > > > and > > > > > > > >>>>>>>>> documentation exists, but making it more prominent or > > more > > > > easily > > > > > > > >>>>>>>> accessed > > > > > > > >>>>>>>>> could help? > > > > > > > >>>>>>>>>> * Extension registry will solve all the world’s > > problems > > > > > > > >>>> (related > > > > > > > >>>>>> to > > > > > > > >>>>>>>>> bundling and build time) > > > > > > > >>>>>>>>>> * Not sure about this one — I don’t know if it’s > > because > > > > > > > >>>> they’re > > > > > > > >>>>>> not > > > > > > > >>>>>>>>> subscribed, their mail client is blocking them, etc. > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>>> I’ve said my bit, now I am eager to hear from other > > > > community > > > > > > > >>>>>> members > > > > > > > >>>>>>>> on > > > > > > > >>>>>>>>> their experiences, steps that helped them, and > > suggestions > > > > for > > > > > > > >>>> the > > > > > > > >>>>>>> future > > > > > > > >>>>>>>>> to continue to make the NiFi community welcoming to new > > > > users. > > > > > > > >>>>>> Thanks. > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>>> Andy LoPresto > > > > > > > >>>>>>>>>> alopre...@apache.org > > > > > > > >>>>>>>>>> alopresto.apa...@gmail.com > > > > > > > >>>>>>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E > > F65B > > > > 2F7D > > > > > > > >>>> EF69 > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>> > > > > > > > >>>>>> > > > > > > > >>>> > > > > > > > > > > > > > > -- > > > Sent from Gmail Mobile > >
