Re: Future-proofing custom bundles that use the Record API

[Bryan Bende]

The record API is part of nifi-record-serialization-services which
gets bundled with nifi-standard-services-api. I think when we get the
extension registry working we should be able to disconnect the
releases of nifi framework from all of the other bundles, instead of
the current situation where every time we do a release all of the
bundles get released even if there are no new changes to the bundle.
So hopefully it would make it easier to upgrade nifi itself without
having to touch the other bundles.

Besides that I would expect that there is a test/integration
environment where you upgrade nifi and test all your flows with your
custom bundles before upgrading production.

On Sat, Jan 26, 2019 at 7:21 AM Mike Thomsen  wrote:
>
> Any recommendations or best practices for making custom bundles that use
> the record API not tied to a specific version of NiFi? We can obviously
> just rebuild when we do an upgrade, but I'm curious if anyone has worked on
> this so that ops folks can do upgrades without having to coordinate with
> developers just to make sure the custom bundles don't break.
>
> Thanks,
>
> Mike

Re: Lowering the barrier of entry

[Bryan Bende]

What does everyone think about bumping the "Developer" section of the
docs ahead of "Processors" so that it's easier to find?

Here is what it would look like -
https://gist.github.com/bbende/279c983f5c80d4fad1431ae81862060f

I also added links to the "Contributor Guide" and the "Maven Projects"
page since I think it would be helpful to make the Developer section
be the one-stop place people look for development help, although I can
see an argument for not mixing wiki content with the docs content.

One issue would be if we want the docs to be fully usable in an
off-line environment, then linking to the wiki won't work, so we could
remove those links, or convert those pages to be part of the docs now
that they are more stable.

For reference, we already have some links in the docs to the wiki:

https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#supplying-a-contribution

On Sat, Jan 26, 2019 at 10:49 AM Joe Witt  wrote:
>
> ...we can.  but the discussion is about how to both lower the bar and offer
> more routes to the bar.
>
> On Sat, Jan 26, 2019, 10:45 AM Otto Fowler 
> > Why wouldn’t we make the archetypes do this?
> >
> >
> > On January 25, 2019 at 18:04:25, Andy LoPresto (alopre...@apache.org)
> > wrote:
> >
> > James,
> >
> > I’m wondering if a page outlining a toy processor (something like
> > `CountText` or `ReverseContent`) and doing a line-by-line annotation from a
> > developer’s perspective would be helpful. It could be a few sections:
> >
> > 1. How to get to this point
> > * running the maven archetype
> > * choosing the directory to install to
> > * putting the class name in the manifest file
> > * etc.
> > 2. The code
> > * here’s the class
> > * here’s what extending AbstractProcessor gets you, etc. A lot of this is
> > currently in the Developer Guide, but in textbook mode
> > * here’s how to modify some code (i.e. write one line of Java that switches
> > it from straight content pass-through to reversing the text)
> > * here’s how to make a unit test (introduce the TestRunner framework, etc.)
> > 3. Running, building, installing
> > * Run your unit test from the IDE/maven
> > * Build the NAR module
> > * Install the NAR in NiFi lib/ or custom/
> > * Restart NiFi
> > * See the NAR loaded in the log
> > * Deploy the component on the canvas
> >
> > I imagine this being written more conversationally/blog-like than most of
> > our current reference documentation to be used as a split-screen
> > walkthrough. Each section could certainly link to the existing detailed
> > documentation for various topics, like the processor lifecycle, etc.
> >
> > Does this sounds like something that would have helped you?
> >
> > Andy LoPresto
> > alopre...@apache.org
> > alopresto.apa...@gmail.com
> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> >
> > > On Jan 25, 2019, at 1:59 PM, James Srinivasan <
> > james.sriniva...@gmail.com>
> > wrote:
> > >
> > > 9) Oh, and the wiki is a little hard to navigate and the contents rather
> > patchy
> > >
> > > On Fri, 25 Jan 2019 at 21:57, James Srinivasan
> > >  wrote:
> > >>
> > >> As someone relatively new to NiFi dev, here's my £0.02. (Yes, I
> > >> realise I could and possibly should submit PRs :)
> > >>
> > >> 1) I'm used to Java and Maven, so used the archetype. It worked fine,
> > >> it would have been nice it if set up unit tests for me.
> > >> 2) The User and Developer documentation is great and comprehensive.
> > >> Finding the developer docs is a little painful (handful of items at
> > >> the end of a scrolling list of 200+ processors)
> > >> 3) The Developer docs could possibly do with a little more clarity on
> > >> processor lifetime i.e. what is called when ^h^h^h - skimming back
> > >> over the docs, it looks pretty clear now
> > >> 4) Some example code for common operations e.g. getting/setting
> > >> attributes or reading/writing/modifying flowfile content would be
> > >> great.
> > >> 5) When using existing processors for inspiration, best practices
> > >> weren't always clear e.g. some generated properties inside
> > >> getSupportedPropertyDescriptors(), others generated a private static
> > >> list on init and returned that. Such differences are inevitable in a
> > >> large project, but it would be nice to have something blessed to start
> > >> from.
> > >> 6) (Minor niggle - layout of the docs doesn't work great on a phone
> > screen)
> > >> 7) I couldn't find (m?)any docs about the Groovy scripting API, but
> > >> the great blog posts by Matt Burgess and others were invaluable
> > >> 8) In case this all sounds too negative, NiFi is fab!
> > >>
> > >> On Fri, 25 Jan 2019 at 18:47, Andrew Grande  wrote:
> > >>>
> > >>> I am not against the archetype. But we need to spell out every step of
> > the
> > >>> way. I'd like to see a user thinking about their custom logic ASAP
> > rather
> > >>> than fighting the tools to get started. Those steps should be
> > brain-dead,
> > >>> just reflexes, if you know what I mean. 

Re: Lowering the barrier of entry

[Andrew Grande]

+1 it's so obvious it should've always been there :)

As for offline access. Well, every time I showed an offline help section to
every user thry were startled :) tells about awareness. The first hunch is
always to google insread of going to a locally hosted doc, so it won't be
an issue, IMO.

Andrew

On Mon, Jan 28, 2019, 8:13 AM Bryan Bende  wrote:

> What does everyone think about bumping the "Developer" section of the
> docs ahead of "Processors" so that it's easier to find?
>
> Here is what it would look like -
> https://gist.github.com/bbende/279c983f5c80d4fad1431ae81862060f
>
> I also added links to the "Contributor Guide" and the "Maven Projects"
> page since I think it would be helpful to make the Developer section
> be the one-stop place people look for development help, although I can
> see an argument for not mixing wiki content with the docs content.
>
> One issue would be if we want the docs to be fully usable in an
> off-line environment, then linking to the wiki won't work, so we could
> remove those links, or convert those pages to be part of the docs now
> that they are more stable.
>
> For reference, we already have some links in the docs to the wiki:
>
>
> https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#supplying-a-contribution
>
> On Sat, Jan 26, 2019 at 10:49 AM Joe Witt  wrote:
> >
> > ...we can.  but the discussion is about how to both lower the bar and
> offer
> > more routes to the bar.
> >
> > On Sat, Jan 26, 2019, 10:45 AM Otto Fowler  wrote:
> >
> > > Why wouldn’t we make the archetypes do this?
> > >
> > >
> > > On January 25, 2019 at 18:04:25, Andy LoPresto (alopre...@apache.org)
> > > wrote:
> > >
> > > James,
> > >
> > > I’m wondering if a page outlining a toy processor (something like
> > > `CountText` or `ReverseContent`) and doing a line-by-line annotation
> from a
> > > developer’s perspective would be helpful. It could be a few sections:
> > >
> > > 1. How to get to this point
> > > * running the maven archetype
> > > * choosing the directory to install to
> > > * putting the class name in the manifest file
> > > * etc.
> > > 2. The code
> > > * here’s the class
> > > * here’s what extending AbstractProcessor gets you, etc. A lot of this
> is
> > > currently in the Developer Guide, but in textbook mode
> > > * here’s how to modify some code (i.e. write one line of Java that
> switches
> > > it from straight content pass-through to reversing the text)
> > > * here’s how to make a unit test (introduce the TestRunner framework,
> etc.)
> > > 3. Running, building, installing
> > > * Run your unit test from the IDE/maven
> > > * Build the NAR module
> > > * Install the NAR in NiFi lib/ or custom/
> > > * Restart NiFi
> > > * See the NAR loaded in the log
> > > * Deploy the component on the canvas
> > >
> > > I imagine this being written more conversationally/blog-like than most
> of
> > > our current reference documentation to be used as a split-screen
> > > walkthrough. Each section could certainly link to the existing detailed
> > > documentation for various topics, like the processor lifecycle, etc.
> > >
> > > Does this sounds like something that would have helped you?
> > >
> > > Andy LoPresto
> > > alopre...@apache.org
> > > alopresto.apa...@gmail.com
> > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> > >
> > > > On Jan 25, 2019, at 1:59 PM, James Srinivasan <
> > > james.sriniva...@gmail.com>
> > > wrote:
> > > >
> > > > 9) Oh, and the wiki is a little hard to navigate and the contents
> rather
> > > patchy
> > > >
> > > > On Fri, 25 Jan 2019 at 21:57, James Srinivasan
> > > >  wrote:
> > > >>
> > > >> As someone relatively new to NiFi dev, here's my £0.02. (Yes, I
> > > >> realise I could and possibly should submit PRs :)
> > > >>
> > > >> 1) I'm used to Java and Maven, so used the archetype. It worked
> fine,
> > > >> it would have been nice it if set up unit tests for me.
> > > >> 2) The User and Developer documentation is great and comprehensive.
> > > >> Finding the developer docs is a little painful (handful of items at
> > > >> the end of a scrolling list of 200+ processors)
> > > >> 3) The Developer docs could possibly do with a little more clarity
> on
> > > >> processor lifetime i.e. what is called when ^h^h^h - skimming back
> > > >> over the docs, it looks pretty clear now
> > > >> 4) Some example code for common operations e.g. getting/setting
> > > >> attributes or reading/writing/modifying flowfile content would be
> > > >> great.
> > > >> 5) When using existing processors for inspiration, best practices
> > > >> weren't always clear e.g. some generated properties inside
> > > >> getSupportedPropertyDescriptors(), others generated a private static
> > > >> list on init and returned that. Such differences are inevitable in a
> > > >> large project, but it would be nice to have something blessed to
> start
> > > >> from.
> > > >> 6) (Minor niggle - layout of the docs doesn't work great on a phone
> > > screen)
> >

Re: Lowering the barrier of entry

[James Srinivasan]

How about separating out User/Developer/Admin into separate docs?

On Mon, 28 Jan 2019 at 16:13, Bryan Bende  wrote:
>
> What does everyone think about bumping the "Developer" section of the
> docs ahead of "Processors" so that it's easier to find?
>
> Here is what it would look like -
> https://gist.github.com/bbende/279c983f5c80d4fad1431ae81862060f
>
> I also added links to the "Contributor Guide" and the "Maven Projects"
> page since I think it would be helpful to make the Developer section
> be the one-stop place people look for development help, although I can
> see an argument for not mixing wiki content with the docs content.
>
> One issue would be if we want the docs to be fully usable in an
> off-line environment, then linking to the wiki won't work, so we could
> remove those links, or convert those pages to be part of the docs now
> that they are more stable.
>
> For reference, we already have some links in the docs to the wiki:
>
> https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#supplying-a-contribution
>
> On Sat, Jan 26, 2019 at 10:49 AM Joe Witt  wrote:
> >
> > ...we can.  but the discussion is about how to both lower the bar and offer
> > more routes to the bar.
> >
> > On Sat, Jan 26, 2019, 10:45 AM Otto Fowler  >
> > > Why wouldn’t we make the archetypes do this?
> > >
> > >
> > > On January 25, 2019 at 18:04:25, Andy LoPresto (alopre...@apache.org)
> > > wrote:
> > >
> > > James,
> > >
> > > I’m wondering if a page outlining a toy processor (something like
> > > `CountText` or `ReverseContent`) and doing a line-by-line annotation from 
> > > a
> > > developer’s perspective would be helpful. It could be a few sections:
> > >
> > > 1. How to get to this point
> > > * running the maven archetype
> > > * choosing the directory to install to
> > > * putting the class name in the manifest file
> > > * etc.
> > > 2. The code
> > > * here’s the class
> > > * here’s what extending AbstractProcessor gets you, etc. A lot of this is
> > > currently in the Developer Guide, but in textbook mode
> > > * here’s how to modify some code (i.e. write one line of Java that 
> > > switches
> > > it from straight content pass-through to reversing the text)
> > > * here’s how to make a unit test (introduce the TestRunner framework, 
> > > etc.)
> > > 3. Running, building, installing
> > > * Run your unit test from the IDE/maven
> > > * Build the NAR module
> > > * Install the NAR in NiFi lib/ or custom/
> > > * Restart NiFi
> > > * See the NAR loaded in the log
> > > * Deploy the component on the canvas
> > >
> > > I imagine this being written more conversationally/blog-like than most of
> > > our current reference documentation to be used as a split-screen
> > > walkthrough. Each section could certainly link to the existing detailed
> > > documentation for various topics, like the processor lifecycle, etc.
> > >
> > > Does this sounds like something that would have helped you?
> > >
> > > Andy LoPresto
> > > alopre...@apache.org
> > > alopresto.apa...@gmail.com
> > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> > >
> > > > On Jan 25, 2019, at 1:59 PM, James Srinivasan <
> > > james.sriniva...@gmail.com>
> > > wrote:
> > > >
> > > > 9) Oh, and the wiki is a little hard to navigate and the contents rather
> > > patchy
> > > >
> > > > On Fri, 25 Jan 2019 at 21:57, James Srinivasan
> > > >  wrote:
> > > >>
> > > >> As someone relatively new to NiFi dev, here's my £0.02. (Yes, I
> > > >> realise I could and possibly should submit PRs :)
> > > >>
> > > >> 1) I'm used to Java and Maven, so used the archetype. It worked fine,
> > > >> it would have been nice it if set up unit tests for me.
> > > >> 2) The User and Developer documentation is great and comprehensive.
> > > >> Finding the developer docs is a little painful (handful of items at
> > > >> the end of a scrolling list of 200+ processors)
> > > >> 3) The Developer docs could possibly do with a little more clarity on
> > > >> processor lifetime i.e. what is called when ^h^h^h - skimming back
> > > >> over the docs, it looks pretty clear now
> > > >> 4) Some example code for common operations e.g. getting/setting
> > > >> attributes or reading/writing/modifying flowfile content would be
> > > >> great.
> > > >> 5) When using existing processors for inspiration, best practices
> > > >> weren't always clear e.g. some generated properties inside
> > > >> getSupportedPropertyDescriptors(), others generated a private static
> > > >> list on init and returned that. Such differences are inevitable in a
> > > >> large project, but it would be nice to have something blessed to start
> > > >> from.
> > > >> 6) (Minor niggle - layout of the docs doesn't work great on a phone
> > > screen)
> > > >> 7) I couldn't find (m?)any docs about the Groovy scripting API, but
> > > >> the great blog posts by Matt Burgess and others were invaluable
> > > >> 8) In case this all sounds too negative, NiFi is fab!
> > > >>
> > > >> On Fri, 25 Jan 20

Re: Lowering the barrier of entry

[Bryan Bende]

Currently it’s broken into General and Developer, so were you thinking of
splitting General into User and Admin?

On Mon, Jan 28, 2019 at 11:34 AM James Srinivasan <
james.sriniva...@gmail.com> wrote:

> How about separating out User/Developer/Admin into separate docs?
>
> On Mon, 28 Jan 2019 at 16:13, Bryan Bende  wrote:
> >
> > What does everyone think about bumping the "Developer" section of the
> > docs ahead of "Processors" so that it's easier to find?
> >
> > Here is what it would look like -
> > https://gist.github.com/bbende/279c983f5c80d4fad1431ae81862060f
> >
> > I also added links to the "Contributor Guide" and the "Maven Projects"
> > page since I think it would be helpful to make the Developer section
> > be the one-stop place people look for development help, although I can
> > see an argument for not mixing wiki content with the docs content.
> >
> > One issue would be if we want the docs to be fully usable in an
> > off-line environment, then linking to the wiki won't work, so we could
> > remove those links, or convert those pages to be part of the docs now
> > that they are more stable.
> >
> > For reference, we already have some links in the docs to the wiki:
> >
> >
> https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#supplying-a-contribution
> >
> > On Sat, Jan 26, 2019 at 10:49 AM Joe Witt  wrote:
> > >
> > > ...we can.  but the discussion is about how to both lower the bar and
> offer
> > > more routes to the bar.
> > >
> > > On Sat, Jan 26, 2019, 10:45 AM Otto Fowler  wrote:
> > >
> > > > Why wouldn’t we make the archetypes do this?
> > > >
> > > >
> > > > On January 25, 2019 at 18:04:25, Andy LoPresto (alopre...@apache.org
> )
> > > > wrote:
> > > >
> > > > James,
> > > >
> > > > I’m wondering if a page outlining a toy processor (something like
> > > > `CountText` or `ReverseContent`) and doing a line-by-line annotation
> from a
> > > > developer’s perspective would be helpful. It could be a few sections:
> > > >
> > > > 1. How to get to this point
> > > > * running the maven archetype
> > > > * choosing the directory to install to
> > > > * putting the class name in the manifest file
> > > > * etc.
> > > > 2. The code
> > > > * here’s the class
> > > > * here’s what extending AbstractProcessor gets you, etc. A lot of
> this is
> > > > currently in the Developer Guide, but in textbook mode
> > > > * here’s how to modify some code (i.e. write one line of Java that
> switches
> > > > it from straight content pass-through to reversing the text)
> > > > * here’s how to make a unit test (introduce the TestRunner
> framework, etc.)
> > > > 3. Running, building, installing
> > > > * Run your unit test from the IDE/maven
> > > > * Build the NAR module
> > > > * Install the NAR in NiFi lib/ or custom/
> > > > * Restart NiFi
> > > > * See the NAR loaded in the log
> > > > * Deploy the component on the canvas
> > > >
> > > > I imagine this being written more conversationally/blog-like than
> most of
> > > > our current reference documentation to be used as a split-screen
> > > > walkthrough. Each section could certainly link to the existing
> detailed
> > > > documentation for various topics, like the processor lifecycle, etc.
> > > >
> > > > Does this sounds like something that would have helped you?
> > > >
> > > > Andy LoPresto
> > > > alopre...@apache.org
> > > > alopresto.apa...@gmail.com
> > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> > > >
> > > > > On Jan 25, 2019, at 1:59 PM, James Srinivasan <
> > > > james.sriniva...@gmail.com>
> > > > wrote:
> > > > >
> > > > > 9) Oh, and the wiki is a little hard to navigate and the contents
> rather
> > > > patchy
> > > > >
> > > > > On Fri, 25 Jan 2019 at 21:57, James Srinivasan
> > > > >  wrote:
> > > > >>
> > > > >> As someone relatively new to NiFi dev, here's my £0.02. (Yes, I
> > > > >> realise I could and possibly should submit PRs :)
> > > > >>
> > > > >> 1) I'm used to Java and Maven, so used the archetype. It worked
> fine,
> > > > >> it would have been nice it if set up unit tests for me.
> > > > >> 2) The User and Developer documentation is great and
> comprehensive.
> > > > >> Finding the developer docs is a little painful (handful of items
> at
> > > > >> the end of a scrolling list of 200+ processors)
> > > > >> 3) The Developer docs could possibly do with a little more
> clarity on
> > > > >> processor lifetime i.e. what is called when ^h^h^h - skimming back
> > > > >> over the docs, it looks pretty clear now
> > > > >> 4) Some example code for common operations e.g. getting/setting
> > > > >> attributes or reading/writing/modifying flowfile content would be
> > > > >> great.
> > > > >> 5) When using existing processors for inspiration, best practices
> > > > >> weren't always clear e.g. some generated properties inside
> > > > >> getSupportedPropertyDescriptors(), others generated a private
> static
> > > > >> list on init and returned that. Such differences are inevitable
> in a
> > > 

Re: Lowering the barrier of entry

[James Srinivasan]

Rather than lumping all the documentation together in a single huge
doc, I was thinking separate entries in the top bar of the NiFi site
under "Documentation" for something like:

General (Overview & Getting Started)
User (User Guide, Expression Language Guide, Record Path Guide &
detailed Processor Usage)
Admin (Admin Guide)
Developer (All the text currently under Developer)

Breaking it out into multiple top-level headings will hopefully help
people find what they need more quickly e.g. with my Developer hat on,
I don't much care about the details of the FooBarProcessor, whereas
with my User hat on, I really want to know about its parameters and
what they mean. Likewise, a non-admin probably doesn't much care about
certificates etc.

Does this makes sense? What do others think?

On Mon, 28 Jan 2019 at 17:04, Bryan Bende  wrote:
>
> Currently it’s broken into General and Developer, so were you thinking of
> splitting General into User and Admin?
>
> On Mon, Jan 28, 2019 at 11:34 AM James Srinivasan <
> james.sriniva...@gmail.com> wrote:
>
> > How about separating out User/Developer/Admin into separate docs?
> >
> > On Mon, 28 Jan 2019 at 16:13, Bryan Bende  wrote:
> > >
> > > What does everyone think about bumping the "Developer" section of the
> > > docs ahead of "Processors" so that it's easier to find?
> > >
> > > Here is what it would look like -
> > > https://gist.github.com/bbende/279c983f5c80d4fad1431ae81862060f
> > >
> > > I also added links to the "Contributor Guide" and the "Maven Projects"
> > > page since I think it would be helpful to make the Developer section
> > > be the one-stop place people look for development help, although I can
> > > see an argument for not mixing wiki content with the docs content.
> > >
> > > One issue would be if we want the docs to be fully usable in an
> > > off-line environment, then linking to the wiki won't work, so we could
> > > remove those links, or convert those pages to be part of the docs now
> > > that they are more stable.
> > >
> > > For reference, we already have some links in the docs to the wiki:
> > >
> > >
> > https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#supplying-a-contribution
> > >
> > > On Sat, Jan 26, 2019 at 10:49 AM Joe Witt  wrote:
> > > >
> > > > ...we can.  but the discussion is about how to both lower the bar and
> > offer
> > > > more routes to the bar.
> > > >
> > > > On Sat, Jan 26, 2019, 10:45 AM Otto Fowler  > wrote:
> > > >
> > > > > Why wouldn’t we make the archetypes do this?
> > > > >
> > > > >
> > > > > On January 25, 2019 at 18:04:25, Andy LoPresto (alopre...@apache.org
> > )
> > > > > wrote:
> > > > >
> > > > > James,
> > > > >
> > > > > I’m wondering if a page outlining a toy processor (something like
> > > > > `CountText` or `ReverseContent`) and doing a line-by-line annotation
> > from a
> > > > > developer’s perspective would be helpful. It could be a few sections:
> > > > >
> > > > > 1. How to get to this point
> > > > > * running the maven archetype
> > > > > * choosing the directory to install to
> > > > > * putting the class name in the manifest file
> > > > > * etc.
> > > > > 2. The code
> > > > > * here’s the class
> > > > > * here’s what extending AbstractProcessor gets you, etc. A lot of
> > this is
> > > > > currently in the Developer Guide, but in textbook mode
> > > > > * here’s how to modify some code (i.e. write one line of Java that
> > switches
> > > > > it from straight content pass-through to reversing the text)
> > > > > * here’s how to make a unit test (introduce the TestRunner
> > framework, etc.)
> > > > > 3. Running, building, installing
> > > > > * Run your unit test from the IDE/maven
> > > > > * Build the NAR module
> > > > > * Install the NAR in NiFi lib/ or custom/
> > > > > * Restart NiFi
> > > > > * See the NAR loaded in the log
> > > > > * Deploy the component on the canvas
> > > > >
> > > > > I imagine this being written more conversationally/blog-like than
> > most of
> > > > > our current reference documentation to be used as a split-screen
> > > > > walkthrough. Each section could certainly link to the existing
> > detailed
> > > > > documentation for various topics, like the processor lifecycle, etc.
> > > > >
> > > > > Does this sounds like something that would have helped you?
> > > > >
> > > > > Andy LoPresto
> > > > > alopre...@apache.org
> > > > > alopresto.apa...@gmail.com
> > > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> > > > >
> > > > > > On Jan 25, 2019, at 1:59 PM, James Srinivasan <
> > > > > james.sriniva...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > 9) Oh, and the wiki is a little hard to navigate and the contents
> > rather
> > > > > patchy
> > > > > >
> > > > > > On Fri, 25 Jan 2019 at 21:57, James Srinivasan
> > > > > >  wrote:
> > > > > >>
> > > > > >> As someone relatively new to NiFi dev, here's my £0.02. (Yes, I
> > > > > >> realise I could and possibly should submit PRs :)
> > > > > >>
> > > > > >> 1) I'm

Re: Lowering the barrier of entry

[Andrew Grande]

Not to throw in a monkey wrench, but does it really make sense to split
User and Developer? In all years I've never seen a user who wasn't a
developer.

Maybe call it a User and Extending NiFi sections?

On Mon, Jan 28, 2019, 9:18 AM James Srinivasan 
wrote:

> Rather than lumping all the documentation together in a single huge
> doc, I was thinking separate entries in the top bar of the NiFi site
> under "Documentation" for something like:
>
> General (Overview & Getting Started)
> User (User Guide, Expression Language Guide, Record Path Guide &
> detailed Processor Usage)
> Admin (Admin Guide)
> Developer (All the text currently under Developer)
>
> Breaking it out into multiple top-level headings will hopefully help
> people find what they need more quickly e.g. with my Developer hat on,
> I don't much care about the details of the FooBarProcessor, whereas
> with my User hat on, I really want to know about its parameters and
> what they mean. Likewise, a non-admin probably doesn't much care about
> certificates etc.
>
> Does this makes sense? What do others think?
>
> On Mon, 28 Jan 2019 at 17:04, Bryan Bende  wrote:
> >
> > Currently it’s broken into General and Developer, so were you thinking of
> > splitting General into User and Admin?
> >
> > On Mon, Jan 28, 2019 at 11:34 AM James Srinivasan <
> > james.sriniva...@gmail.com> wrote:
> >
> > > How about separating out User/Developer/Admin into separate docs?
> > >
> > > On Mon, 28 Jan 2019 at 16:13, Bryan Bende  wrote:
> > > >
> > > > What does everyone think about bumping the "Developer" section of the
> > > > docs ahead of "Processors" so that it's easier to find?
> > > >
> > > > Here is what it would look like -
> > > > https://gist.github.com/bbende/279c983f5c80d4fad1431ae81862060f
> > > >
> > > > I also added links to the "Contributor Guide" and the "Maven
> Projects"
> > > > page since I think it would be helpful to make the Developer section
> > > > be the one-stop place people look for development help, although I
> can
> > > > see an argument for not mixing wiki content with the docs content.
> > > >
> > > > One issue would be if we want the docs to be fully usable in an
> > > > off-line environment, then linking to the wiki won't work, so we
> could
> > > > remove those links, or convert those pages to be part of the docs now
> > > > that they are more stable.
> > > >
> > > > For reference, we already have some links in the docs to the wiki:
> > > >
> > > >
> > >
> https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#supplying-a-contribution
> > > >
> > > > On Sat, Jan 26, 2019 at 10:49 AM Joe Witt 
> wrote:
> > > > >
> > > > > ...we can.  but the discussion is about how to both lower the bar
> and
> > > offer
> > > > > more routes to the bar.
> > > > >
> > > > > On Sat, Jan 26, 2019, 10:45 AM Otto Fowler <
> ottobackwa...@gmail.com
> > > wrote:
> > > > >
> > > > > > Why wouldn’t we make the archetypes do this?
> > > > > >
> > > > > >
> > > > > > On January 25, 2019 at 18:04:25, Andy LoPresto (
> alopre...@apache.org
> > > )
> > > > > > wrote:
> > > > > >
> > > > > > James,
> > > > > >
> > > > > > I’m wondering if a page outlining a toy processor (something like
> > > > > > `CountText` or `ReverseContent`) and doing a line-by-line
> annotation
> > > from a
> > > > > > developer’s perspective would be helpful. It could be a few
> sections:
> > > > > >
> > > > > > 1. How to get to this point
> > > > > > * running the maven archetype
> > > > > > * choosing the directory to install to
> > > > > > * putting the class name in the manifest file
> > > > > > * etc.
> > > > > > 2. The code
> > > > > > * here’s the class
> > > > > > * here’s what extending AbstractProcessor gets you, etc. A lot of
> > > this is
> > > > > > currently in the Developer Guide, but in textbook mode
> > > > > > * here’s how to modify some code (i.e. write one line of Java
> that
> > > switches
> > > > > > it from straight content pass-through to reversing the text)
> > > > > > * here’s how to make a unit test (introduce the TestRunner
> > > framework, etc.)
> > > > > > 3. Running, building, installing
> > > > > > * Run your unit test from the IDE/maven
> > > > > > * Build the NAR module
> > > > > > * Install the NAR in NiFi lib/ or custom/
> > > > > > * Restart NiFi
> > > > > > * See the NAR loaded in the log
> > > > > > * Deploy the component on the canvas
> > > > > >
> > > > > > I imagine this being written more conversationally/blog-like than
> > > most of
> > > > > > our current reference documentation to be used as a split-screen
> > > > > > walkthrough. Each section could certainly link to the existing
> > > detailed
> > > > > > documentation for various topics, like the processor lifecycle,
> etc.
> > > > > >
> > > > > > Does this sounds like something that would have helped you?
> > > > > >
> > > > > > Andy LoPresto
> > > > > > alopre...@apache.org
> > > > > > alopresto.apa...@gmail.com
> > > > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B

Re: Lowering the barrier of entry

[James Srinivasan]

Throw away :-)

In our org, we have some people who can use NiFi to create flows with
the built-in processors. I guess HWX would all them "Data Flow
Managers".
They might know some scripting, so can write Jython scripts.

We have far fewer (approx 1) person (me) who can write custom
processors in Java. Sometimes, I also create flows.

I'd like to have lots more of both kinds of people.

"User" & "Extending NiFi" actually maps to the internal training I
run, so I'd be happy :-)

James

On Mon, 28 Jan 2019 at 18:01, Andrew Grande  wrote:
>
> Not to throw in a monkey wrench, but does it really make sense to split
> User and Developer? In all years I've never seen a user who wasn't a
> developer.
>
> Maybe call it a User and Extending NiFi sections?
>
> On Mon, Jan 28, 2019, 9:18 AM James Srinivasan 
> wrote:
>
> > Rather than lumping all the documentation together in a single huge
> > doc, I was thinking separate entries in the top bar of the NiFi site
> > under "Documentation" for something like:
> >
> > General (Overview & Getting Started)
> > User (User Guide, Expression Language Guide, Record Path Guide &
> > detailed Processor Usage)
> > Admin (Admin Guide)
> > Developer (All the text currently under Developer)
> >
> > Breaking it out into multiple top-level headings will hopefully help
> > people find what they need more quickly e.g. with my Developer hat on,
> > I don't much care about the details of the FooBarProcessor, whereas
> > with my User hat on, I really want to know about its parameters and
> > what they mean. Likewise, a non-admin probably doesn't much care about
> > certificates etc.
> >
> > Does this makes sense? What do others think?
> >
> > On Mon, 28 Jan 2019 at 17:04, Bryan Bende  wrote:
> > >
> > > Currently it’s broken into General and Developer, so were you thinking of
> > > splitting General into User and Admin?
> > >
> > > On Mon, Jan 28, 2019 at 11:34 AM James Srinivasan <
> > > james.sriniva...@gmail.com> wrote:
> > >
> > > > How about separating out User/Developer/Admin into separate docs?
> > > >
> > > > On Mon, 28 Jan 2019 at 16:13, Bryan Bende  wrote:
> > > > >
> > > > > What does everyone think about bumping the "Developer" section of the
> > > > > docs ahead of "Processors" so that it's easier to find?
> > > > >
> > > > > Here is what it would look like -
> > > > > https://gist.github.com/bbende/279c983f5c80d4fad1431ae81862060f
> > > > >
> > > > > I also added links to the "Contributor Guide" and the "Maven
> > Projects"
> > > > > page since I think it would be helpful to make the Developer section
> > > > > be the one-stop place people look for development help, although I
> > can
> > > > > see an argument for not mixing wiki content with the docs content.
> > > > >
> > > > > One issue would be if we want the docs to be fully usable in an
> > > > > off-line environment, then linking to the wiki won't work, so we
> > could
> > > > > remove those links, or convert those pages to be part of the docs now
> > > > > that they are more stable.
> > > > >
> > > > > For reference, we already have some links in the docs to the wiki:
> > > > >
> > > > >
> > > >
> > https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#supplying-a-contribution
> > > > >
> > > > > On Sat, Jan 26, 2019 at 10:49 AM Joe Witt 
> > wrote:
> > > > > >
> > > > > > ...we can.  but the discussion is about how to both lower the bar
> > and
> > > > offer
> > > > > > more routes to the bar.
> > > > > >
> > > > > > On Sat, Jan 26, 2019, 10:45 AM Otto Fowler <
> > ottobackwa...@gmail.com
> > > > wrote:
> > > > > >
> > > > > > > Why wouldn’t we make the archetypes do this?
> > > > > > >
> > > > > > >
> > > > > > > On January 25, 2019 at 18:04:25, Andy LoPresto (
> > alopre...@apache.org
> > > > )
> > > > > > > wrote:
> > > > > > >
> > > > > > > James,
> > > > > > >
> > > > > > > I’m wondering if a page outlining a toy processor (something like
> > > > > > > `CountText` or `ReverseContent`) and doing a line-by-line
> > annotation
> > > > from a
> > > > > > > developer’s perspective would be helpful. It could be a few
> > sections:
> > > > > > >
> > > > > > > 1. How to get to this point
> > > > > > > * running the maven archetype
> > > > > > > * choosing the directory to install to
> > > > > > > * putting the class name in the manifest file
> > > > > > > * etc.
> > > > > > > 2. The code
> > > > > > > * here’s the class
> > > > > > > * here’s what extending AbstractProcessor gets you, etc. A lot of
> > > > this is
> > > > > > > currently in the Developer Guide, but in textbook mode
> > > > > > > * here’s how to modify some code (i.e. write one line of Java
> > that
> > > > switches
> > > > > > > it from straight content pass-through to reversing the text)
> > > > > > > * here’s how to make a unit test (introduce the TestRunner
> > > > framework, etc.)
> > > > > > > 3. Running, building, installing
> > > > > > > * Run your unit test from the IDE/maven
> > > > > > > * Build the NAR module
> > > > > > > * Install

Re: Lowering the barrier of entry

[Andy LoPresto]

There are many, *many* people who use NiFi and never write a line of code. 
Please feel free to peruse the us...@nifi.apache.org 
 list for examples of these (see also: origins of 
NiFi). Separating the User Guide (how to interact with the application via the 
UI and API) and the Developer Guide (how to extend/change/and understand the 
behavior of NiFi by viewing/modifying code) target very different (while 
sometimes overlapping) audiences and I feel strongly should stay separated to 
reduce the likelihood of overwhelming either group with unnecessary information 
and making it harder to find what they are looking for. 

I think moving the Developer Guide link up to the top is a good idea, and there 
has definitely been discussion about this in the past. I can remember no 
objections. 

I don’t know that additional menu items on the main site would help, because 
from the landing page in the documentation section, those titles (User, Admin, 
and hopefully now Developer) are broken out immediately. 

I would vote against including wiki pages as part of the top-level 
documentation. I think linking to them from content in a guide is helpful, but 
as Bryan pointed out, they change frequently and are not deployed with the 
application. I think if we decide there is information in the wiki that is 
stable and valuable, it should be moved into top-level (i.e. static generated) 
documentation and hosted there & deployed with the application. In my opinion, 
the information in the wiki should be frequently changing or under current 
discussion/proposed. There is greater access to modifying that info, but that 
should be fixed by encouraging the community to make PRs to the versioned 
documentation when appropriate. 

Examples:
* feature proposals, process FAQs, release notes, migration guidance, 
articles = wiki
* component documentation, user/dev/admin guides, 
walkthroughs/reference documents = documentation as code in NiFi project proper



Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Jan 28, 2019, at 10:46 AM, James Srinivasan  
> wrote:
> 
> Throw away :-)
> 
> In our org, we have some people who can use NiFi to create flows with
> the built-in processors. I guess HWX would all them "Data Flow
> Managers".
> They might know some scripting, so can write Jython scripts.
> 
> We have far fewer (approx 1) person (me) who can write custom
> processors in Java. Sometimes, I also create flows.
> 
> I'd like to have lots more of both kinds of people.
> 
> "User" & "Extending NiFi" actually maps to the internal training I
> run, so I'd be happy :-)
> 
> James
> 
> On Mon, 28 Jan 2019 at 18:01, Andrew Grande  wrote:
>> 
>> Not to throw in a monkey wrench, but does it really make sense to split
>> User and Developer? In all years I've never seen a user who wasn't a
>> developer.
>> 
>> Maybe call it a User and Extending NiFi sections?
>> 
>> On Mon, Jan 28, 2019, 9:18 AM James Srinivasan 
>> wrote:
>> 
>>> Rather than lumping all the documentation together in a single huge
>>> doc, I was thinking separate entries in the top bar of the NiFi site
>>> under "Documentation" for something like:
>>> 
>>> General (Overview & Getting Started)
>>> User (User Guide, Expression Language Guide, Record Path Guide &
>>> detailed Processor Usage)
>>> Admin (Admin Guide)
>>> Developer (All the text currently under Developer)
>>> 
>>> Breaking it out into multiple top-level headings will hopefully help
>>> people find what they need more quickly e.g. with my Developer hat on,
>>> I don't much care about the details of the FooBarProcessor, whereas
>>> with my User hat on, I really want to know about its parameters and
>>> what they mean. Likewise, a non-admin probably doesn't much care about
>>> certificates etc.
>>> 
>>> Does this makes sense? What do others think?
>>> 
>>> On Mon, 28 Jan 2019 at 17:04, Bryan Bende  wrote:
 
 Currently it’s broken into General and Developer, so were you thinking of
 splitting General into User and Admin?
 
 On Mon, Jan 28, 2019 at 11:34 AM James Srinivasan <
 james.sriniva...@gmail.com> wrote:
 
> How about separating out User/Developer/Admin into separate docs?
> 
> On Mon, 28 Jan 2019 at 16:13, Bryan Bende  wrote:
>> 
>> What does everyone think about bumping the "Developer" section of the
>> docs ahead of "Processors" so that it's easier to find?
>> 
>> Here is what it would look like -
>> https://gist.github.com/bbende/279c983f5c80d4fad1431ae81862060f
>> 
>> I also added links to the "Contributor Guide" and the "Maven
>>> Projects"
>> page since I think it would be helpful to make the Developer section
>> be the one-stop place people look for development help, although I
>>> can
>> see an argument for not mixing wiki content with the docs content.
>> 
>> One issue would be