Re: Lowering the barrier of entry
...we can. but the discussion is about how to both lower the bar and offer more routes to the bar. On Sat, Jan 26, 2019, 10:45 AM Otto Fowler Why wouldn’t we make the archetypes do this? > > > On January 25, 2019 at 18:04:25, Andy LoPresto (alopre...@apache.org) > wrote: > > James, > > I’m wondering if a page outlining a toy processor (something like > `CountText` or `ReverseContent`) and doing a line-by-line annotation from a > developer’s perspective would be helpful. It could be a few sections: > > 1. How to get to this point > * running the maven archetype > * choosing the directory to install to > * putting the class name in the manifest file > * etc. > 2. The code > * here’s the class > * here’s what extending AbstractProcessor gets you, etc. A lot of this is > currently in the Developer Guide, but in textbook mode > * here’s how to modify some code (i.e. write one line of Java that switches > it from straight content pass-through to reversing the text) > * here’s how to make a unit test (introduce the TestRunner framework, etc.) > 3. Running, building, installing > * Run your unit test from the IDE/maven > * Build the NAR module > * Install the NAR in NiFi lib/ or custom/ > * Restart NiFi > * See the NAR loaded in the log > * Deploy the component on the canvas > > I imagine this being written more conversationally/blog-like than most of > our current reference documentation to be used as a split-screen > walkthrough. Each section could certainly link to the existing detailed > documentation for various topics, like the processor lifecycle, etc. > > Does this sounds like something that would have helped you? > > Andy LoPresto > alopre...@apache.org > alopresto.apa...@gmail.com > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > On Jan 25, 2019, at 1:59 PM, James Srinivasan < > james.sriniva...@gmail.com> > wrote: > > > > 9) Oh, and the wiki is a little hard to navigate and the contents rather > patchy > > > > On Fri, 25 Jan 2019 at 21:57, James Srinivasan > > wrote: > >> > >> As someone relatively new to NiFi dev, here's my £0.02. (Yes, I > >> realise I could and possibly should submit PRs :) > >> > >> 1) I'm used to Java and Maven, so used the archetype. It worked fine, > >> it would have been nice it if set up unit tests for me. > >> 2) The User and Developer documentation is great and comprehensive. > >> Finding the developer docs is a little painful (handful of items at > >> the end of a scrolling list of 200+ processors) > >> 3) The Developer docs could possibly do with a little more clarity on > >> processor lifetime i.e. what is called when ^h^h^h - skimming back > >> over the docs, it looks pretty clear now > >> 4) Some example code for common operations e.g. getting/setting > >> attributes or reading/writing/modifying flowfile content would be > >> great. > >> 5) When using existing processors for inspiration, best practices > >> weren't always clear e.g. some generated properties inside > >> getSupportedPropertyDescriptors(), others generated a private static > >> list on init and returned that. Such differences are inevitable in a > >> large project, but it would be nice to have something blessed to start > >> from. > >> 6) (Minor niggle - layout of the docs doesn't work great on a phone > screen) > >> 7) I couldn't find (m?)any docs about the Groovy scripting API, but > >> the great blog posts by Matt Burgess and others were invaluable > >> 8) In case this all sounds too negative, NiFi is fab! > >> > >> On Fri, 25 Jan 2019 at 18:47, Andrew Grande wrote: > >>> > >>> I am not against the archetype. But we need to spell out every step of > the > >>> way. I'd like to see a user thinking about their custom logic ASAP > rather > >>> than fighting the tools to get started. Those steps should be > brain-dead, > >>> just reflexes, if you know what I mean. Hell, let them create a custom > >>> processor project or prototype in a script by accident even! :) > >>> > >>> On Fri, Jan 25, 2019, 10:43 AM Bryan Bende wrote: > >>> > That makes sense about the best practice for deploying to an > additional lib directory. > > So for the project structure you are saying it would be easier to have > a repo somewhere with essentially the same thing that is in the > archetype, but they just clone it and rename it themselves (what the > archetype does for you)? > > Something that I think would be awesome is if we could provide a > web-based project initializer that would essentially run the archetype > behind the scenes and then let you download the archive of the code, > just like the spring-boot starter [1]. Not sure if their initializr is > something that can be re-used and customized [2]. > > The problem is we would need to host that somewhere. > > [1] https://start.spring.io/ > [2] https://github.com/spring-io/initializr > > On Fri, Jan 25, 2019 at 12:56 PM Andrew Grande > wrote:
Re: Lowering the barrier of entry
Why wouldn’t we make the archetypes do this? On January 25, 2019 at 18:04:25, Andy LoPresto (alopre...@apache.org) wrote: James, I’m wondering if a page outlining a toy processor (something like `CountText` or `ReverseContent`) and doing a line-by-line annotation from a developer’s perspective would be helpful. It could be a few sections: 1. How to get to this point * running the maven archetype * choosing the directory to install to * putting the class name in the manifest file * etc. 2. The code * here’s the class * here’s what extending AbstractProcessor gets you, etc. A lot of this is currently in the Developer Guide, but in textbook mode * here’s how to modify some code (i.e. write one line of Java that switches it from straight content pass-through to reversing the text) * here’s how to make a unit test (introduce the TestRunner framework, etc.) 3. Running, building, installing * Run your unit test from the IDE/maven * Build the NAR module * Install the NAR in NiFi lib/ or custom/ * Restart NiFi * See the NAR loaded in the log * Deploy the component on the canvas I imagine this being written more conversationally/blog-like than most of our current reference documentation to be used as a split-screen walkthrough. Each section could certainly link to the existing detailed documentation for various topics, like the processor lifecycle, etc. Does this sounds like something that would have helped you? Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jan 25, 2019, at 1:59 PM, James Srinivasan wrote: > > 9) Oh, and the wiki is a little hard to navigate and the contents rather patchy > > On Fri, 25 Jan 2019 at 21:57, James Srinivasan > wrote: >> >> As someone relatively new to NiFi dev, here's my £0.02. (Yes, I >> realise I could and possibly should submit PRs :) >> >> 1) I'm used to Java and Maven, so used the archetype. It worked fine, >> it would have been nice it if set up unit tests for me. >> 2) The User and Developer documentation is great and comprehensive. >> Finding the developer docs is a little painful (handful of items at >> the end of a scrolling list of 200+ processors) >> 3) The Developer docs could possibly do with a little more clarity on >> processor lifetime i.e. what is called when ^h^h^h - skimming back >> over the docs, it looks pretty clear now >> 4) Some example code for common operations e.g. getting/setting >> attributes or reading/writing/modifying flowfile content would be >> great. >> 5) When using existing processors for inspiration, best practices >> weren't always clear e.g. some generated properties inside >> getSupportedPropertyDescriptors(), others generated a private static >> list on init and returned that. Such differences are inevitable in a >> large project, but it would be nice to have something blessed to start >> from. >> 6) (Minor niggle - layout of the docs doesn't work great on a phone screen) >> 7) I couldn't find (m?)any docs about the Groovy scripting API, but >> the great blog posts by Matt Burgess and others were invaluable >> 8) In case this all sounds too negative, NiFi is fab! >> >> On Fri, 25 Jan 2019 at 18:47, Andrew Grande wrote: >>> >>> I am not against the archetype. But we need to spell out every step of the >>> way. I'd like to see a user thinking about their custom logic ASAP rather >>> than fighting the tools to get started. Those steps should be brain-dead, >>> just reflexes, if you know what I mean. Hell, let them create a custom >>> processor project or prototype in a script by accident even! :) >>> >>> On Fri, Jan 25, 2019, 10:43 AM Bryan Bende wrote: >>> That makes sense about the best practice for deploying to an additional lib directory. So for the project structure you are saying it would be easier to have a repo somewhere with essentially the same thing that is in the archetype, but they just clone it and rename it themselves (what the archetype does for you)? Something that I think would be awesome is if we could provide a web-based project initializer that would essentially run the archetype behind the scenes and then let you download the archive of the code, just like the spring-boot starter [1]. Not sure if their initializr is something that can be re-used and customized [2]. The problem is we would need to host that somewhere. [1] https://start.spring.io/ [2] https://github.com/spring-io/initializr On Fri, Jan 25, 2019 at 12:56 PM Andrew Grande wrote: > > We assume they create new projects from archetypes every day. They don't. > > We also assume they know how to deploy new NARs. Most don't. Especially if > we want them to follow best practices and create an additional NAR bundles > directory entry im the config (vs dumping into nifi lib). > > I can attest that I feel a bit lost myself every
Re: Lowering the barrier of entry
That would have been great - the only things I would add are reading/writing attributes & reading the processor's properties. Maybe something about testing in NiFi using GenerateFlowFile too? I ended up referring to the bundled processors a lot, but it was sometimes hard to see the wood for the trees. On Fri, 25 Jan 2019 at 23:04, Andy LoPresto wrote: > > James, > > I’m wondering if a page outlining a toy processor (something like `CountText` > or `ReverseContent`) and doing a line-by-line annotation from a developer’s > perspective would be helpful. It could be a few sections: > > 1. How to get to this point > * running the maven archetype > * choosing the directory to install to > * putting the class name in the manifest file > * etc. > 2. The code > * here’s the class > * here’s what extending AbstractProcessor gets you, etc. A lot of > this is currently in the Developer Guide, but in textbook mode > * here’s how to modify some code (i.e. write one line of Java that > switches it from straight content pass-through to reversing the text) > * here’s how to make a unit test (introduce the TestRunner framework, > etc.) > 3. Running, building, installing > * Run your unit test from the IDE/maven > * Build the NAR module > * Install the NAR in NiFi lib/ or custom/ > * Restart NiFi > * See the NAR loaded in the log > * Deploy the component on the canvas > > I imagine this being written more conversationally/blog-like than most of our > current reference documentation to be used as a split-screen walkthrough. > Each section could certainly link to the existing detailed documentation for > various topics, like the processor lifecycle, etc. > > Does this sounds like something that would have helped you? > > Andy LoPresto > alopre...@apache.org > alopresto.apa...@gmail.com > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > On Jan 25, 2019, at 1:59 PM, James Srinivasan > > wrote: > > > > 9) Oh, and the wiki is a little hard to navigate and the contents rather > > patchy > > > > On Fri, 25 Jan 2019 at 21:57, James Srinivasan > > wrote: > >> > >> As someone relatively new to NiFi dev, here's my £0.02. (Yes, I > >> realise I could and possibly should submit PRs :) > >> > >> 1) I'm used to Java and Maven, so used the archetype. It worked fine, > >> it would have been nice it if set up unit tests for me. > >> 2) The User and Developer documentation is great and comprehensive. > >> Finding the developer docs is a little painful (handful of items at > >> the end of a scrolling list of 200+ processors) > >> 3) The Developer docs could possibly do with a little more clarity on > >> processor lifetime i.e. what is called when ^h^h^h - skimming back > >> over the docs, it looks pretty clear now > >> 4) Some example code for common operations e.g. getting/setting > >> attributes or reading/writing/modifying flowfile content would be > >> great. > >> 5) When using existing processors for inspiration, best practices > >> weren't always clear e.g. some generated properties inside > >> getSupportedPropertyDescriptors(), others generated a private static > >> list on init and returned that. Such differences are inevitable in a > >> large project, but it would be nice to have something blessed to start > >> from. > >> 6) (Minor niggle - layout of the docs doesn't work great on a phone screen) > >> 7) I couldn't find (m?)any docs about the Groovy scripting API, but > >> the great blog posts by Matt Burgess and others were invaluable > >> 8) In case this all sounds too negative, NiFi is fab! > >> > >> On Fri, 25 Jan 2019 at 18:47, Andrew Grande wrote: > >>> > >>> I am not against the archetype. But we need to spell out every step of the > >>> way. I'd like to see a user thinking about their custom logic ASAP rather > >>> than fighting the tools to get started. Those steps should be brain-dead, > >>> just reflexes, if you know what I mean. Hell, let them create a custom > >>> processor project or prototype in a script by accident even! :) > >>> > >>> On Fri, Jan 25, 2019, 10:43 AM Bryan Bende wrote: > >>> > That makes sense about the best practice for deploying to an > additional lib directory. > > So for the project structure you are saying it would be easier to have > a repo somewhere with essentially the same thing that is in the > archetype, but they just clone it and rename it themselves (what the > archetype does for you)? > > Something that I think would be awesome is if we could provide a > web-based project initializer that would essentially run the archetype > behind the scenes and then let you download the archive of the code, > just like the spring-boot starter [1]. Not sure if their initializr is > something that can be re-used and customized [2]. > > The problem is we would need
Future-proofing custom bundles that use the Record API
Any recommendations or best practices for making custom bundles that use the record API not tied to a specific version of NiFi? We can obviously just rebuild when we do an upgrade, but I'm curious if anyone has worked on this so that ops folks can do upgrades without having to coordinate with developers just to make sure the custom bundles don't break. Thanks, Mike
