Rather than lumping all the documentation together in a single huge doc, I was thinking separate entries in the top bar of the NiFi site under "Documentation" for something like:
General (Overview & Getting Started) User (User Guide, Expression Language Guide, Record Path Guide & detailed Processor Usage) Admin (Admin Guide) Developer (All the text currently under Developer) Breaking it out into multiple top-level headings will hopefully help people find what they need more quickly e.g. with my Developer hat on, I don't much care about the details of the FooBarProcessor, whereas with my User hat on, I really want to know about its parameters and what they mean. Likewise, a non-admin probably doesn't much care about certificates etc. Does this makes sense? What do others think? On Mon, 28 Jan 2019 at 17:04, Bryan Bende <bbe...@gmail.com> wrote: > > Currently it’s broken into General and Developer, so were you thinking of > splitting General into User and Admin? > > On Mon, Jan 28, 2019 at 11:34 AM James Srinivasan < > james.sriniva...@gmail.com> wrote: > > > How about separating out User/Developer/Admin into separate docs? > > > > On Mon, 28 Jan 2019 at 16:13, Bryan Bende <bbe...@gmail.com> wrote: > > > > > > What does everyone think about bumping the "Developer" section of the > > > docs ahead of "Processors" so that it's easier to find? > > > > > > Here is what it would look like - > > > https://gist.github.com/bbende/279c983f5c80d4fad1431ae81862060f > > > > > > I also added links to the "Contributor Guide" and the "Maven Projects" > > > page since I think it would be helpful to make the Developer section > > > be the one-stop place people look for development help, although I can > > > see an argument for not mixing wiki content with the docs content. > > > > > > One issue would be if we want the docs to be fully usable in an > > > off-line environment, then linking to the wiki won't work, so we could > > > remove those links, or convert those pages to be part of the docs now > > > that they are more stable. > > > > > > For reference, we already have some links in the docs to the wiki: > > > > > > > > https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#supplying-a-contribution > > > > > > On Sat, Jan 26, 2019 at 10:49 AM Joe Witt <joe.w...@gmail.com> wrote: > > > > > > > > ...we can. but the discussion is about how to both lower the bar and > > offer > > > > more routes to the bar. > > > > > > > > On Sat, Jan 26, 2019, 10:45 AM Otto Fowler <ottobackwa...@gmail.com > > wrote: > > > > > > > > > Why wouldn’t we make the archetypes do this? > > > > > > > > > > > > > > > On January 25, 2019 at 18:04:25, Andy LoPresto (alopre...@apache.org > > ) > > > > > wrote: > > > > > > > > > > James, > > > > > > > > > > I’m wondering if a page outlining a toy processor (something like > > > > > `CountText` or `ReverseContent`) and doing a line-by-line annotation > > from a > > > > > developer’s perspective would be helpful. It could be a few sections: > > > > > > > > > > 1. How to get to this point > > > > > * running the maven archetype > > > > > * choosing the directory to install to > > > > > * putting the class name in the manifest file > > > > > * etc. > > > > > 2. The code > > > > > * here’s the class > > > > > * here’s what extending AbstractProcessor gets you, etc. A lot of > > this is > > > > > currently in the Developer Guide, but in textbook mode > > > > > * here’s how to modify some code (i.e. write one line of Java that > > switches > > > > > it from straight content pass-through to reversing the text) > > > > > * here’s how to make a unit test (introduce the TestRunner > > framework, etc.) > > > > > 3. Running, building, installing > > > > > * Run your unit test from the IDE/maven > > > > > * Build the NAR module > > > > > * Install the NAR in NiFi lib/ or custom/ > > > > > * Restart NiFi > > > > > * See the NAR loaded in the log > > > > > * Deploy the component on the canvas > > > > > > > > > > I imagine this being written more conversationally/blog-like than > > most of > > > > > our current reference documentation to be used as a split-screen > > > > > walkthrough. Each section could certainly link to the existing > > detailed > > > > > documentation for various topics, like the processor lifecycle, etc. > > > > > > > > > > Does this sounds like something that would have helped you? > > > > > > > > > > Andy LoPresto > > > > > alopre...@apache.org > > > > > alopresto.apa...@gmail.com > > > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > > > > > > > > > On Jan 25, 2019, at 1:59 PM, James Srinivasan < > > > > > james.sriniva...@gmail.com> > > > > > wrote: > > > > > > > > > > > > 9) Oh, and the wiki is a little hard to navigate and the contents > > rather > > > > > patchy > > > > > > > > > > > > On Fri, 25 Jan 2019 at 21:57, James Srinivasan > > > > > > <james.sriniva...@gmail.com> wrote: > > > > > >> > > > > > >> As someone relatively new to NiFi dev, here's my £0.02. (Yes, I > > > > > >> realise I could and possibly should submit PRs :) > > > > > >> > > > > > >> 1) I'm used to Java and Maven, so used the archetype. It worked > > fine, > > > > > >> it would have been nice it if set up unit tests for me. > > > > > >> 2) The User and Developer documentation is great and > > comprehensive. > > > > > >> Finding the developer docs is a little painful (handful of items > > at > > > > > >> the end of a scrolling list of 200+ processors) > > > > > >> 3) The Developer docs could possibly do with a little more > > clarity on > > > > > >> processor lifetime i.e. what is called when ^h^h^h - skimming back > > > > > >> over the docs, it looks pretty clear now > > > > > >> 4) Some example code for common operations e.g. getting/setting > > > > > >> attributes or reading/writing/modifying flowfile content would be > > > > > >> great. > > > > > >> 5) When using existing processors for inspiration, best practices > > > > > >> weren't always clear e.g. some generated properties inside > > > > > >> getSupportedPropertyDescriptors(), others generated a private > > static > > > > > >> list on init and returned that. Such differences are inevitable > > in a > > > > > >> large project, but it would be nice to have something blessed to > > start > > > > > >> from. > > > > > >> 6) (Minor niggle - layout of the docs doesn't work great on a > > phone > > > > > screen) > > > > > >> 7) I couldn't find (m?)any docs about the Groovy scripting API, > > but > > > > > >> the great blog posts by Matt Burgess and others were invaluable > > > > > >> 8) In case this all sounds too negative, NiFi is fab! > > > > > >> > > > > > >> On Fri, 25 Jan 2019 at 18:47, Andrew Grande <apere...@gmail.com> > > wrote: > > > > > >>> > > > > > >>> I am not against the archetype. But we need to spell out every > > step of > > > > > the > > > > > >>> way. I'd like to see a user thinking about their custom logic > > ASAP > > > > > rather > > > > > >>> than fighting the tools to get started. Those steps should be > > > > > brain-dead, > > > > > >>> just reflexes, if you know what I mean. Hell, let them create a > > custom > > > > > >>> processor project or prototype in a script by accident even! :) > > > > > >>> > > > > > >>> On Fri, Jan 25, 2019, 10:43 AM Bryan Bende <bbe...@gmail.com> > > wrote: > > > > > >>> > > > > > >>>> That makes sense about the best practice for deploying to an > > > > > >>>> additional lib directory. > > > > > >>>> > > > > > >>>> So for the project structure you are saying it would be easier > > to have > > > > > >>>> a repo somewhere with essentially the same thing that is in the > > > > > >>>> archetype, but they just clone it and rename it themselves > > (what the > > > > > >>>> archetype does for you)? > > > > > >>>> > > > > > >>>> Something that I think would be awesome is if we could provide a > > > > > >>>> web-based project initializer that would essentially run the > > archetype > > > > > >>>> behind the scenes and then let you download the archive of the > > code, > > > > > >>>> just like the spring-boot starter [1]. Not sure if their > > initializr is > > > > > >>>> something that can be re-used and customized [2]. > > > > > >>>> > > > > > >>>> The problem is we would need to host that somewhere. > > > > > >>>> > > > > > >>>> [1] https://start.spring.io/ > > > > > >>>> [2] https://github.com/spring-io/initializr > > > > > >>>> > > > > > >>>> On Fri, Jan 25, 2019 at 12:56 PM Andrew Grande < > > apere...@gmail.com> > > > > > wrote: > > > > > >>>>> > > > > > >>>>> We assume they create new projects from archetypes every day. > > They > > > > > don't. > > > > > >>>>> > > > > > >>>>> We also assume they know how to deploy new NARs. Most don't. > > > > > Especially > > > > > >>>> if > > > > > >>>>> we want them to follow best practices and create an additional > > NAR > > > > > >>>> bundles > > > > > >>>>> directory entry im the config (vs dumping into nifi lib). > > > > > >>>>> > > > > > >>>>> I can attest that I feel a bit lost myself every time I need > > to come > > > > > back > > > > > >>>>> to this and refresh my brain synapses. If we could make these > > not > > > > > require > > > > > >>>>> any of that and make simple thinga dead simple.... > > > > > >>>>> > > > > > >>>>> Andrew > > > > > >>>>> > > > > > >>>>> On Fri, Jan 25, 2019, 9:47 AM Bryan Bende <bbe...@gmail.com> > > wrote: > > > > > >>>>> > > > > > >>>>>> Andrew, > > > > > >>>>>> > > > > > >>>>>> I'm not disagreeing with your points, but I'm curious how you > > see > > > > > >>>>>> those two ideas being different from the processor archetype > > and the > > > > > >>>>>> wiki page with the archetype commands? > > > > > >>>>>> > > > > > >>>>>> Is it just that people don't know about it? > > > > > >>>>>> > > > > > >>>>>> -Bryan > > > > > >>>>>> > > > > > >>>>>> [1] > > > > > >>>>>> > > > > > >>>> > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions > > > > > >>>>>> > > > > > >>>>>> On Fri, Jan 25, 2019 at 12:23 PM Otto Fowler < > > > > > ottobackwa...@gmail.com> > > > > > > > > > > >>>>>> wrote: > > > > > >>>>>>> > > > > > >>>>>>> I think this ties into my other discuss thread on refreshing > > the > > > > > >>>>>> archetypes > > > > > >>>>>>> > > > > > >>>>>>> > > > > > >>>>>>> On January 25, 2019 at 11:50:10, Andrew Grande ( > > apere...@gmail.com > > > > > ) > > > > > >>>>>> wrote: > > > > > >>>>>>> > > > > > >>>>>>> I consistently see my users struggling when they move up the > > nifi > > > > > >>>> food > > > > > >>>>>>> chain and start looking at custom processors. The good > > content > > > > > about > > > > > >>>>>>> prototyping processsors via scripting processors and > > finalizing > > > > > with > > > > > >>>> a > > > > > >>>>>> full > > > > > >>>>>>> NAR bundle is everywhere but where it should be. > > > > > >>>>>>> > > > > > >>>>>>> A few simple changes could help (not *more* docs). They are > > great, > > > > > >>>> much > > > > > >>>>>>> better than in many other projecta, but people are already > > drowning > > > > > >>>> in > > > > > >>>>>>> those. > > > > > >>>>>>> > > > > > >>>>>>> How about: > > > > > >>>>>>> > > > > > >>>>>>> + ISP has a pre-populated processor sceleton. A simple no-op > > to > > > > > fill > > > > > >>>> in > > > > > >>>>>> is > > > > > >>>>>>> miles better than a blank text area (which invokes a blank > > stare). > > > > > >>>>>>> > > > > > >>>>>>> + As much as we may loook down on this, but... A simple > > guide to a > > > > > >>>> full > > > > > >>>>>> NAR > > > > > >>>>>>> build as a series of copy/paste commands. > > > > > >>>>>>> > > > > > >>>>>>> There's more, but this should fit the context for now. > > > > > >>>>>>> > > > > > >>>>>>> Andrew > > > > > >>>>>>> > > > > > >>>>>>> On Fri, Jan 25, 2019, 8:13 AM Mike Thomsen < > > mikerthom...@gmail.com > > > > > > > > > > > >>>>>> wrote: > > > > > >>>>>>> > > > > > >>>>>>>> One of the changes we should make is to create a separate > > guide > > > > > for > > > > > >>>>>>> product > > > > > >>>>>>>> vendors on how to build and maintain a bundle. We're at > > that point > > > > > >>>>>> where > > > > > >>>>>>>> vendors will have to do it on their own as extension > > providers, so > > > > > >>>> it > > > > > >>>>>>> would > > > > > >>>>>>>> be very helpful for them to have a simple and straight > > forward > > > > > >>>> document > > > > > >>>>>>>> showing them what should be there, best practices for > > > > > >>>> maintainability > > > > > >>>>>> and > > > > > >>>>>>>> where to announce it. > > > > > >>>>>>>> > > > > > >>>>>>>> On Fri, Jan 25, 2019 at 9:59 AM Bryan Bende < > > bbe...@gmail.com> > > > > > >>>> wrote: > > > > > >>>>>>>> > > > > > >>>>>>>>> I think we have a lot more documentation than most > > projects, but > > > > > >>>> I > > > > > >>>>>>>>> think an issue is that content is scattered in many > > different > > > > > >>>>>>>>> locations, and some of the docs are huge reference guides > > where > > > > > >>>> it > > > > > >>>>>> can > > > > > >>>>>>>>> be hard to find all the pieces of what you are trying to > > do. > > > > > >>>>>>>>> > > > > > >>>>>>>>> The first thing a new contributor wants to do is get the > > code > > > > > >>>> and run > > > > > >>>>>>>>> a build, and we do have a quick-start guide linked to on > > the > > > > > >>>> site, > > > > > >>>>>> but > > > > > >>>>>>>>> I think there is a lot of extra information in there that > > is not > > > > > >>>>>>>>> really relevant to someone just wanting get the code and > > build. > > > > > >>>> We > > > > > >>>>>>>>> could have separate guides per OS like "Build NiFi on > > Linux", > > > > > >>>> "Build > > > > > >>>>>>>>> NiFi on Windows", etc, where each guide was 4-5 steps like: > > > > > >>>>>>>>> > > > > > >>>>>>>>> - Clone repo > > > > > >>>>>>>>> - checkout master > > > > > >>>>>>>>> - run maven > > > > > >>>>>>>>> - cd to assembly > > > > > >>>>>>>>> - ./bin/nifi.sh > > > > > >>>>>>>>> > > > > > >>>>>>>>> The next thing they want to do is contribute a change, and > > we > > > > > >>>> have a > > > > > >>>>>>>>> great contributor guide, but again I think there could be > > a very > > > > > >>>>>> short > > > > > >>>>>>>>> tutorial for the most common steps: > > > > > >>>>>>>>> > > > > > >>>>>>>>> - fork repo > > > > > >>>>>>>>> - clone fork > > > > > >>>>>>>>> - create branch > > > > > >>>>>>>>> - make changes > > > > > >>>>>>>>> - push branch > > > > > >>>>>>>>> - submit pr > > > > > >>>>>>>>> > > > > > >>>>>>>>> and then say something like "for a more detailed > > description of > > > > > >>>> the > > > > > >>>>>>>>> contribution process, please reference the Contributor > > Guide". > > > > > >>>>>>>>> > > > > > >>>>>>>>> If we then make these getting started guides more prominent > > > > > >>>> right in > > > > > >>>>>>>>> the middle of the NiFi homepage, then maybe they will be > > easier > > > > > >>>> to > > > > > >>>>>>>>> find for new community members. > > > > > >>>>>>>>> > > > > > >>>>>>>>> We can keep extending this idea to other common tasks > > beyond just > > > > > >>>>>>>>> building and contributing. > > > > > >>>>>>>>> > > > > > >>>>>>>>> > > > > > >>>>>>>>> On Thu, Jan 24, 2019 at 8:03 PM Andy LoPresto < > > > > > >>>> alopre...@apache.org> > > > > > >>>>>>>>> wrote: > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> Hi folks, > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> Based on some recent (and long-term) experiences, I > > wanted to > > > > > >>>>>> discuss > > > > > >>>>>>>>> with the community what we could do to lower the barrier of > > > > > >>>> entry to > > > > > >>>>>>>> using > > > > > >>>>>>>>> & contributing to NiFi. I hope to get some good feedback > > from > > > > > >>>> both > > > > > >>>>>>>>> long-time and newer members, and determine some immediate > > > > > >>>> concrete > > > > > >>>>>>> steps > > > > > >>>>>>>> we > > > > > >>>>>>>>> can take. > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> Problems identified: > > > > > >>>>>>>>>> * NiFi has a number of custom profiles, so a simple “mvn > > clean > > > > > >>>>>>> install” > > > > > >>>>>>>>> in project root doesn’t get a new developer up and running > > > > > >>>>>> immediately > > > > > >>>>>>>>>> * The API is very well defined, but for new contributors, > > it > > > > > >>>> can > > > > > >>>>>> be a > > > > > >>>>>>>>> challenge to know where to put functionality, and building > > a > > > > > >>>> custom > > > > > >>>>>>>>> processor + NAR and deploying isn’t a one-step process > > > > > >>>>>>>>>> * Project size (and build size/time) is large. This can > > > > > >>>> restrict > > > > > >>>>>> the > > > > > >>>>>>>>> minimum hardware necessary, elongate the development > > cycle, etc. > > > > > >>>>>>>>>> * Some new users do not receive mailing list replies > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> Possible solutions: > > > > > >>>>>>>>>> * On a clean git clone, “mvn clean install” should build a > > > > > >>>> working > > > > > >>>>>>>>> instance. Maybe we provide a quickstart.sh script to > > handle the > > > > > >>>>>> default > > > > > >>>>>>>>> maven build, change to the target directory, and start > > NiFi? > > > > > >>>>>>>>>> * Individual contributors have written excellent blogs, > > and > > > > > >>>>>>>>> documentation exists, but making it more prominent or more > > easily > > > > > >>>>>>>> accessed > > > > > >>>>>>>>> could help? > > > > > >>>>>>>>>> * Extension registry will solve all the world’s problems > > > > > >>>> (related > > > > > >>>>>> to > > > > > >>>>>>>>> bundling and build time) > > > > > >>>>>>>>>> * Not sure about this one — I don’t know if it’s because > > > > > >>>> they’re > > > > > >>>>>> not > > > > > >>>>>>>>> subscribed, their mail client is blocking them, etc. > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> I’ve said my bit, now I am eager to hear from other > > community > > > > > >>>>>> members > > > > > >>>>>>>> on > > > > > >>>>>>>>> their experiences, steps that helped them, and suggestions > > for > > > > > >>>> the > > > > > >>>>>>> future > > > > > >>>>>>>>> to continue to make the NiFi community welcoming to new > > users. > > > > > >>>>>> Thanks. > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> Andy LoPresto > > > > > >>>>>>>>>> alopre...@apache.org > > > > > >>>>>>>>>> alopresto.apa...@gmail.com > > > > > >>>>>>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B > > 2F7D > > > > > >>>> EF69 > > > > > >>>>>>>>>> > > > > > >>>>>>>>> > > > > > >>>>>>>> > > > > > >>>>>> > > > > > >>>> > > > > > > > > -- > Sent from Gmail Mobile
