Re: [DISCUSS] - Markdown option for documentation artifacts

[dan bress]

+1

On Wed, Jun 8, 2016 at 7:05 AM Andre  wrote:

> +1 on this + a template that matches existing additional.html
> On 8 Jun 2016 04:28, "Bryan Rosander"  wrote:
>
> > Hey all,
> >
> > When writing documentation (e.g. the additionalDetails.html for a
> > processor) it would be nice to have the option to use Markdown instead of
> > html.
> >
> > I think Markdown is easier to read and write than raw HTML and for simple
> > cases does the job pretty well.  It also has the advantage of being able
> to
> > be translated into other document types easily and it would be rendered
> by
> > default in Github when the file is clicked.
> >
> > There is an MIT-licensed Markdown maven plugin (
> > https://github.com/walokra/markdown-page-generator-plugin) that seems
> like
> > it might work for translating additionalDetails.md (and others) into an
> > equivalent html page.
> >
> > Thanks,
> > Bryan Rosander
> >
>

Re: [ANNOUNCE] New Apache NiFi Committer James Wing

[dan bress]

Welcome James!

On Wed, May 11, 2016 at 1:15 PM Mark Payne  wrote:

> Welcome, James! Glad to have you aboard!
>
> > On May 11, 2016, at 4:10 PM, Joe Witt  wrote:
> >
> > On behalf of the Apache NiFi PMC, I am pleased to announce that James
> > Wing has accepted the PMC's inviation to become a committer on the
> > Apache NiFi project.  James has been an excellent contributor to a
> > number of code submissions, peer reviews, mailing list discussions and
> > decision making and it is greatly appreciated.  We look forward to his
> > continued involvement in the project.
> >
> > Welcome and congratulations!
>
>

Re: @DynamicProperty annotation of more than one property?

[dan bress]

Russell,
   I agree the syntax for multiple DynamicProperties is a little awkward,
and that our documentation could be improved in this situation(I didn't see
reference to @DynamicProperty in the developer doc).

   Also Java8 supports the concept of repeatable annotations[1], which
would erase the need for the @DynamicProperties by allowing you to specify
multiple @DynamicProperty annotations on a single class.  Since NiFi
appears to be running on Java8, I think we should create a ticket to update
the documentation generation code to handle repeated annotations, and
update our processors/controller services/reporting tasks to use them.  We
should also create a ticket to update the developer documentation to
describe the annotations usage.

[1] http://docs.oracle.com/javase/tutorial/java/annotations/repeating.html

Dan

On Tue, Apr 19, 2016 at 7:23 AM Russell Bateman <
russell.bate...@perfectsearchcorp.com> wrote:

> Thanks, Joe. This worked.
>
> You know, I'm just the sort of guy that would add stuff like that to the
> documentation if the way were smoothed to it. The problem as I see it
> with NiFi doc is that it's really good the way it is and if we add
> more--and we should--it's going to get longer and harder to read.
> Someone should slightly redesign it with links to subordinate documents
> to avoid making it too complicated yet give folk a place to go for
> deeper help. I would be happy to help out once such a thing as that is
> decided.
>
> Russ
>
> On 04/15/2016 06:39 PM, Joe Percivall wrote:
> > Hello Russell,
> >
> > The annotation you are looking for is @DynamicProperties[1] an example
> of it in use is in the PutFTP processor[2].
> >
> > [1]
> https://github.com/apache/nifi/blob/e4b7e47836edf47042973e604005058c28eed23b/nifi-api/src/main/java/org/apache/nifi/annotation/behavior/DynamicProperties.java
> > [2]
> https://github.com/apache/nifi/blob/e4b7e47836edf47042973e604005058c28eed23b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutFTP.java#L50
> >
> >
> > Hope that helps,
> > Joe
> >
> > - - - - - -
> > Joseph Percivall
> > linkedin.com/in/Percivall
> > e: joeperciv...@yahoo.com
> >
> >
> >
> > On Friday, April 15, 2016 7:46 PM, Russell Bateman <
> russell.bate...@perfectsearchcorp.com> wrote:
> >
> >
> >
> > What's the syntax for defining more than one dynamic property for a
> > processor? I need to specify up to three distinctly different ones and
> > attempting to do it all in
> >
> > @DynamicProperty(
> >   name = "{blah,blah2,blah3}",
> >   value = "{\"blah-value\",\"blah2-value\",\"blah3-value\"}",
> > supportsExpressionLanguage = false,
> >   description = "Big blah, blah, blah..."
> > )
> >
> > is under-powered.
> >
> > Many thanks.
>
>

Re: Your Distro

[dan bress]

Paul,
   Have you tried joining the mailing list using the instructions here:
https://nifi.apache.org/mailing_lists.html

I know in the past, we have had issues with people who aren't
subscribed being unable to send to the list.  This seems weird to me as
well, but since you are bringing up some concerns and making suggestions it
might be best to join the list so that they can be heard and addressed.

Dan

On Wed, Mar 23, 2016 at 9:41 AM Matthew Clarke 
wrote:

> -- Forwarded message --
> From: Paul Nahay 
> Date: Wed, Mar 23, 2016 at 10:31 AM
> Subject: Your Distro
> To: Matthew Clarke 
>
>
> This is the 2nd time I've gotten an email I sent to
> dev@nifi.apache.org
> returned to me:
>
>
>
> Hi. This is the qmail-send program at apache.org.
> I'm afraid I wasn't able to deliver your message to the following
> addresses.
> This is a permanent error; I've given up. Sorry it didn't work out.
>
> :
> ezmlm-reject: fatal: Sorry, I don't accept messages of MIME
> Content-Type 'text/html' (#5.2.3)
>

Re: Re: Processor additional documentation

[dan bress]

Uwe,
   No its not a problem to have both index.html and additionalDetails.html
 The NiFi framework generates nearly all of the documentation for your
processor for you.  It will generate information about the properties and
relationships your processor exposes to its users.  If you need to express
more about your processor, then that is where additionalDetails.html comes
into play.  For example, if your processor uses a custom query language.

Generated index.html example:
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.attributes.UpdateAttribute/index.html

additionalDetails.html example:
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.attributes.UpdateAttribute/additionalDetails.html

On Fri, Mar 18, 2016 at 10:54 AM Uwe Geercken  wrote:

> Bryan,
>
> all looks ok. I looked into the nifi-home/work/docs folder. There is
> nothing but a components folder. Inside there is a folder for my processor:
> com.datamelt.nifi.test.TemplateProcessor and inside the folder there is a
> file index.html and it contains the code of my additionalDetails.html file.
>
> when I open the file in the web browser it looks good. I looked at other
> index.html files and they look similar.
>
> but I noted that some folders have an inde.html file AND an
> additionalDetails.html file. maybe that is the problem?
>
> greetings,
>
> Uwe
>
>
>
> Gesendet: Freitag, 18. März 2016 um 16:18 Uhr
> Von: "Bryan Bende" 
> An: dev@nifi.apache.org
> Betreff: Re: Processor additional documentation
> Hi Uwe,
>
> Do you have the additionalDetails.html file in your processors jar project,
> under src/main/resources?
>
> Similar to this:
>
> https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/src/main/resources
>
> The expected project structure is described here:
>
> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-ExampleProcessorBundleStructure[https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-ExampleProcessorBundleStructure]
> 
>
> If you think that part is setup correctly, can you check under
> nifi_home/work/docs and see if com.datamelt.nifi.test.TemplateProcessor is
> there?
>
> -Bryan
>
> On Fri, Mar 18, 2016 at 11:04 AM, Uwe Geercken 
> wrote:
>
> >
> > Hello,
> >
> > I am writing my first processor. As described in the documentation, I
> have
> > added an HTML file to be used when the user selects "Usage":
> >
> > docs/com.datamelt.nifi.test.TemplateProcessor/additionalDetails.html
> >
> > This is located in the root or the Processors nar file.
> >
> > The processor class is this:
> >
> > com/datamelt/nifi/test/TemplateProcessor.class
> >
> > The processor works, but selecting "Usage" won't show my HTML file.
> >
> > I understood that I write the HTML file and Nifi will picks it up when it
> > starts. Or is this not true?
> >
> > Thanks for feedback,
> >
> > Uwe
> >
>

Re: Re: Re: Processor additional documentation

[dan bress]

Uwe,
   No, the index.html is generated for you.  additionalDetails.html is your
responsibility only if you feel like the generated index.html doesn't fully
describe your processor.

   I would guess 80% of the included processors do not have
additionalDetails.html.  If you haven't browsed here [1] at examples of the
generated index.html and user supplied additionalDetails.html, it might
clear things up.

[1] https://nifi.apache.org/docs.html

Dan

On Fri, Mar 18, 2016 at 11:08 AM Uwe Geercken <uwe.geerc...@web.de> wrote:

> Dan,
>
> but maybe I have a wrong understanding: do I have to create an index.html
> file? Currently I have only created an additionalDetails.html file.
>
> I will also try to reduce the html code to a minimum and see if it is a
> problem with my code.
>
> Bye,
>
> Uwe
>
> > Gesendet: Freitag, 18. März 2016 um 19:03 Uhr
> > Von: "dan bress" <danbr...@gmail.com>
> > An: dev@nifi.apache.org
> > Betreff: Re: Re: Processor additional documentation
> >
> > Uwe,
> >No its not a problem to have both index.html and
> additionalDetails.html
> >  The NiFi framework generates nearly all of the documentation for your
> > processor for you.  It will generate information about the properties and
> > relationships your processor exposes to its users.  If you need to
> express
> > more about your processor, then that is where additionalDetails.html
> comes
> > into play.  For example, if your processor uses a custom query language.
> >
> > Generated index.html example:
> >
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.attributes.UpdateAttribute/index.html
> >
> > additionalDetails.html example:
> >
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.attributes.UpdateAttribute/additionalDetails.html
> >
> > On Fri, Mar 18, 2016 at 10:54 AM Uwe Geercken <uwe.geerc...@web.de>
> wrote:
> >
> > > Bryan,
> > >
> > > all looks ok. I looked into the nifi-home/work/docs folder. There is
> > > nothing but a components folder. Inside there is a folder for my
> processor:
> > > com.datamelt.nifi.test.TemplateProcessor and inside the folder there
> is a
> > > file index.html and it contains the code of my additionalDetails.html
> file.
> > >
> > > when I open the file in the web browser it looks good. I looked at
> other
> > > index.html files and they look similar.
> > >
> > > but I noted that some folders have an inde.html file AND an
> > > additionalDetails.html file. maybe that is the problem?
> > >
> > > greetings,
> > >
> > > Uwe
> > >
> > >
> > >
> > > Gesendet: Freitag, 18. März 2016 um 16:18 Uhr
> > > Von: "Bryan Bende" <bbe...@gmail.com>
> > > An: dev@nifi.apache.org
> > > Betreff: Re: Processor additional documentation
> > > Hi Uwe,
> > >
> > > Do you have the additionalDetails.html file in your processors jar
> project,
> > > under src/main/resources?
> > >
> > > Similar to this:
> > >
> > >
> https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/src/main/resources
> > >
> > > The expected project structure is described here:
> > >
> > >
> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-ExampleProcessorBundleStructure[https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-ExampleProcessorBundleStructure]
> <https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-ExampleProcessorBundleStructure[https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions%23MavenProjectsforExtensions-ExampleProcessorBundleStructure]>
> > > <
> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-ExampleProcessorBundleStructure[https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions%23MavenProjectsforExtensions-ExampleProcessorBundleStructure]
> >
> > >
> > > If you think that part is setup correctly, can you check under
> > > nifi_home/work/docs and see if
> com.datamelt.nifi.test.TemplateProcessor is
> > > there?
> > >
> > > -Bryan
> > >
> > > On Fri, Mar 18, 2016 at 11:04 AM, Uwe Geercken <uwe.geerc...@web.de>
> > > wrote:
> > >
> > > >
> > > > Hello,
> > > >
> > > > I am writing my first processor. As described in the documentation, I
> > > have
> > > > added an HTML file to be used when the user selects "Usage":
> > > >
> > > > docs/com.datamelt.nifi.test.TemplateProcessor/additionalDetails.html
> > > >
> > > > This is located in the root or the Processors nar file.
> > > >
> > > > The processor class is this:
> > > >
> > > > com/datamelt/nifi/test/TemplateProcessor.class
> > > >
> > > > The processor works, but selecting "Usage" won't show my HTML file.
> > > >
> > > > I understood that I write the HTML file and Nifi will picks it up
> when it
> > > > starts. Or is this not true?
> > > >
> > > > Thanks for feedback,
> > > >
> > > > Uwe
> > > >
> > >
> >
>

Re: Are we thinking about Penalization all wrong?

[dan bress]

Mark,
   I agree with all the points you mention about penalization being
confusing, and I think the ability to apply a penalty to Flowfile's outside
of a processor is a clearer way to express what is happening.

   I worry that having the penalty be a property of the connection would
also be confusing.  To me, penalizing a FlowFile is an action you do to a
FlowFile.  In my head, connections don't do actions on FlowFile, they just
sort them and move them along.  I might find it confusing that the
connection is "doing things" to the flow files, unless there was some kind
of visual cue as to what was going on.  Kind of like how people have
brought up that the "expire" concept is a little confusing, because of the
lack of visual cue.

So when I started typing this email I was thinking we should have a new
concept of a "penalizer" that's kind of like a processor but just puts a
penalty on a flow file.  After typing it, that might be a new construct
that isn't really needed, and I'm OK with this being put on a connection,
but I would like there to be a visual cue on the connection indicating that
it is penalizing flow files.

On Thu, Jan 28, 2016 at 8:34 AM Mark Payne  wrote:

> All,
>
> I've been thinking about how we handle the concept of penalizing
> FlowFiles. We've had a lot of questions
> lately about how penalization works & the concept in general. Seems the
> following problems exist:
>
> - Confusion about difference between penalization & yielding
> - DFM sees option to configure penalization period on all processors, even
> if they don't penalize FlowFiles.
> - DFM cannot set penalty duration in 1 case and set a different value for
> a different case (different relationship, for example).
> - Developers often forget to call penalize()
> - Developer has to determine whether or not to penalize when building a
> processor. It is based on what the developer will
> think may make sense, but in reality DFM's sometimes want to penalize
> things when the processor doesn't behave that way.
>
> I'm wondering if it doesn't make sense to remove the concept of
> penalization all together from Processors and instead
> move the Penalty Duration so that it's a setting on the Connection. I
> think this would clear up the confusion and give the DFM
> more control over when/how long to penalize. Could set to the default to
> 30 seconds for self-looping connections and no penalization
> for other connections.
>
> Any thoughts?
>
> Thanks
> -Mark

Re: Auto-Organize Layout

[dan bress]

Maybe not exactly "auto-layout" but I would back a notion of having the
components snap to a coarser grain grid than what we currently have.
Sometimes I care a lot about having everything line up in the graph
horizontally and vertically, and it always takes a long time to achieve
this.

I could see this being achieved by snapping the component to the same spot
horizontally as the component above it when you move it underneath another
component.  Or some magical "auto snap" button that does its best to align
everything with its nearest neighbors.

On Tue, Jan 19, 2016 at 12:37 PM Ryan H  wrote:

> I like your idea Rob, that would help with lining up relationships too
> (straight lines).
>
> On Matt's note, I don't think there should be a "standard" either, although
> best practices are always out there.
>
> On Matt's note of putting failures up above processes, we do that too.
> Totally depends on who made the flow first.  Sometimes, people don't even
> follow a convention in the same flow.xml file.
>
> For these reasons, I'd recommend alternate views to the flow.
>
> We have a couple projects that just allow you to rearrange a node-based
> graph, based on your preference, hierarchy, circular, pyramid, etc.
>
> Applying this to NiFi, having a couple different default auto-layout
> options that you can swap your current view to, but NOT change the original
> flow, would be nice.
>
> It would let you walk into someone else's, potentially large, dataflow and
> have a familiar way to view the flow.
>
> Ryan
>
>
> On Tue, Jan 19, 2016 at 2:03 PM, Rob Moran  wrote:
>
> > I agree with Matt's points. I was just replying with something similar
> > basically saying I think trying to set a standard would not be
> > well-received.
> >
> > I believe what could be more useful are layout tools that would help
> users
> > place components to help achieve their preferred layouts. For example,
> the
> > ability to align (left, right, center) components
> > or horizontally/vertically distribute components evenly. Other features
> > such as snap-to and/or smart-guides could make it easier for users to
> > follow their organization's best practices when designing a flow.
> >
> > Rob
> >
> > On Tue, Jan 19, 2016 at 1:49 PM, Matthew Clarke <
> matt.clarke@gmail.com
> > >
> > wrote:
> >
> > > Ryan,
> > >
> > >   Setting a standard is a difficult thing to do.  The
> complexity
> > > that can exist in many flows would make enforcing a standard difficult.
> > The
> > > first example you provide of success to points right while failures
> point
> > > up is not recommended. It would be better to have failures point down
> > since
> > > it is common to put labels over processor(s). Any relationships
> pointing
> > up
> > > would pass through these labels making both the relationship box and
> the
> > > label hard to read.  It is often coomon to see flows designed with a
> > > combination of left to right and top to bottom design.
> > >
> > > Matt
> > >
> > > On Tue, Jan 19, 2016 at 12:07 PM, Ryan H 
> > > wrote:
> > >
> > > > Hi Rob,
> > > > Yea we did, it was at the end of the meeting.
> > > >
> > > > I think it would be useful to have a couple default type views
> that
> > > > help standardize flow layout across the community.
> > > >
> > > > For example, when we organize processors left-to-right, failure
> > > > relationships always point up, and success always point right.
> > > > Alternatively, when we organize processors up-and-down, failure
> > > > relationships always point left, and successes always point down.
> > > >
> > > > Of course, in some of these scenarios there are processors that
> > have
> > > > more than 1 success relationship, but that's when a good layout
> library
> > > > would come into play.
> > > >
> > > > What do you think?
> > > >
> > > > On Tue, Jan 19, 2016 at 11:10 AM, Rob Moran 
> wrote:
> > > >
> > > > > Ryan - I think we spoke briefly (at a very high level) about this
> at
> > a
> > > > > prior meetup. What alternate views did you have in mind, and in
> what
> > > ways
> > > > > do you think they'd be useful?
> > > > >
> > > > > Rob
> > > > >
> > > > > On Tue, Jan 19, 2016 at 10:56 AM, Ryan H <
> > rhendrickson.w...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > It'd be pretty awesome if NiFi provided the ability to
> > auto-organize
> > > a
> > > > > > layout.
> > > > > >
> > > > > > Maybe even just a auto-organized layout that doesn't change the
> > > > flow.xml,
> > > > > > just an alternate view.
> > > > > >
> > > > > > Looking at these demos here: http://js.cytoscape.org/#demos
> > > > > >
> > > > > > Ryan
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Component documentation improvements

[dan bress]

I just tossed some comments on the wikipage.

TL;DR I took a look at generating docs as part of the build months ago.  I
think its doable, and someone should pursue it.

On Mon, Jan 18, 2016 at 2:38 PM Joe Witt  wrote:

> Oleg,
>
> "Yes, there will be breaking changes. Its not a question of IF, but
> rather WHEN."
>
> I disagree.  It is always a question of IF.
>
> We have to be extremely judicious in the use of breaking changes and
> we owe the user/developer based excellent justification in such cases.
>
> I will comment on the Wiki page for the substance of this particular
> proposal (documentation generation).
>
> Joe
>
> On Mon, Jan 18, 2016 at 1:22 PM, Oleg Zhurakousky
>  wrote:
> > Josh
> >
> > FWIW, let’s use WIKI comments to maintain a discussion. It will be
> simpler in the end to compile a resolution and move on.
> >
> > Yet, I’ll reply here anyway.
> > Yes, there will be breaking changes. Its not a question of IF, but
> rather WHEN.
> > What we can do is make it less painful by introducing certain changes
> gradually with deprecations and clear communication to the community on
> what is about to change. Other mechanics could be applied here as well, but
> before we get into the mechanics, I’d like to see if there are any more
> ideas, concerns etc, so we can have a join resolution as to what is a
> sustainable documentation model of the future NiFi, then we can figure out
> how to get there.
> >
> > Cheers
> > Oleg
> >
> >> On Jan 18, 2016, at 1:08 PM, Joshua Davis 
> wrote:
> >>
> >> Oleg,
> >>
> >> Interesting document, what impact would it have on existing
> installations
> >> of NIFI?
> >>
> >> What would be the upgrade path for Custom Processors?
> >>
> >> Are we breaking compatibility with the previous way of doing
> documentation?
> >>
> >> Why not create a simple content repository that can hold the
> documentation
> >> information?
> >>
> >> Is there a plan for multiple languages?
> >>
> >> Joshua Davis
> >> Senior Consultant
> >> Hortonworks Professional Services
> >> (407)476-6752
> >>
> >>
> >>
> >>
> >>
> >>
> >> On 1/18/16, 12:09 PM, "Oleg Zhurakousky" 
> >> wrote:
> >>
> >>> Guys
> >>>
> >>> I¹ve just finished initial draft of the proposal to improve our
> component
> >>> documentation mechanisms (e.g., Processors, ControllerServices etc).
> >>>
> https://cwiki.apache.org/confluence/display/NIFI/Component+documentation+i
> >>> mprovements
> >>> Please give it a read and let¹s get discussion going.
> >>>
> >>> Cheers
> >>> Oleg
> >>
> >>
> >
>

Re: [ANNOUNCE] New Apache NiFi Committer Ricky Saltzer

[dan bress]

Welcome Ricky!  Thanks for your contributions and I look forward to you
pushing Apache NiFi forward!

On Wed, Oct 21, 2015 at 3:04 PM Tony Kurc  wrote:

> NiFi Community!
>
> Great news!
>
> On behalf of the Apache NiFI PMC, I am very pleased to announce that Ricky
> Saltzer has accepted the PMC's invitation to become a committer on the
> Apache NiFi project. We greatly appreciate all of Ricky's hard work and
> generous contributions to the project. We look forward to his continued
> involvement in the project.
>
> Welcome Ricky, and congratulations!
>
> Tony
>

Re: "Integration" style Unit Tests

[dan bress]

I agree with what JoeS is saying.   Having these core integration aspects
of the system tested in an automated way only when needed(like before a
release) would be a huge plus for NiFi.

I think Sean provides a great mechanism for implementing this with
Categories.  Sean, can you speak to the build server bit?  Is this
something we have?  Is this something we are working towards?

We would still need a mechanism for capturing the credentials required to
run tests that integrate with remote services.  Maybe this lives on the
build server?  Maybe we have a stub .properties file that shows credentials
you need to interact with AWS, kafka, Twitter, etc..

Also if we can tag TestJdbcHugeStream with @SlowTests so I don't have to
wait for it every time I run tests then I will be t h r i l l e d.

Dan

On Wed, Oct 14, 2015 at 3:19 PM Sean Busbey <bus...@cloudera.com> wrote:

> JUnit Categories would work perfectly for this:
>
> https://github.com/junit-team/junit/wiki/Categories
>
> You can group either at the class test layer or on individual tests.
>
> we can then run the categories that require external resources on
> build servers that have the needed instances present.
>
> On Wed, Oct 14, 2015 at 1:46 PM, Joe Skora <jsk...@gmail.com> wrote:
> > I agree with you that these shouldn't run as normal unit tests.  But, I'm
> > worried that it means this functionality will not be regularly tested as
> > external APIs change.  Mocking an external API doesn't validate much, if
> I
> > don't understand the external resource I'll probably make the same
> mistakes
> > in the mock as I do in the client, so testing against the real external
> > resource is the only way to know things work.
> >
> > Perhaps a set of test scripts are needed to automate groups of @ignored
> > tests that share an external dependency such as an AWS script, a Hadoop
> > script, a Kafka script, etc?
> >
> > I have never used JUnit's filtering capability, but it might be possible
> to
> > use that in combination with a Maven profile to automate those test
> groups
> > instead of requiring scripts be maintained.
> >
> > Just brainstorming.
> >
> > Regards,
> > JoeS
> >
> > On Mon, Oct 12, 2015 at 8:22 PM, Joe Witt <joe.w...@gmail.com> wrote:
> >
> >> Dan,
> >>
> >> Yeah this is a good housekeeping item to bring up.  My view here is
> >> that B is the only answer.  Unit tests should not be calling out to
> >> external services - period.
> >>
> >> Thanks
> >> Joe
> >>
> >> On Mon, Oct 12, 2015 at 8:17 PM, dan bress <danbr...@gmail.com> wrote:
> >> > Devs,
> >> >While working on integrating and testing the work Yu did for
> >> > NIFI-774/DeleteS3Object, I noticed that a few of the unit tests for
> the
> >> > processors in that AWS bundle(Put and Fetch S3) actually interact
> with S3
> >> > directly, and were marked as @Ignore.  If I wanted to un @Ignore them
> and
> >> > actually run them I needed to set up an AWS account, then copy the
> >> > credentials into an aws-credentials.properties file and put that in my
> >> home
> >> > directory to get the tests to pass.  This dashed my hopes of a
> relatively
> >> > simple merge and turned it into a bit of work for me.  I'm not
> blaming Yu
> >> > or anyone for this, just wanted to open up a discussion on better
> ways of
> >> > solving this.
> >> >
> >> > Problem:
> >> > @Ignore'd tests don't get run, probably ever.  Why?  Because running
> them
> >> > is a pain in the butt, I agree with NIFI-438[1] "If tests could talk
> they
> >> > would say don't @Ignore me".  I appreciate that there are special
> >> > circumstances for using this, but it would probably benefit everyone
> if
> >> we
> >> > sure we used it only for the truly special circumstances.
> >> >
> >> >
> >> > Solutions:
> >> > a. Run these tests using the failsafe plugin[2] instead of the
> surefire
> >> > plugin.  This way they get run every time, and if they fail that
> >> > information gets reported but it does not stop the build.
> >> > b. Mock out the service(I appreciate that this may not always be
> >> possible)
> >> > c. Provide instructions somewhere so that someone with no experience
> with
> >> > these processors/tests can run them.
> >> >
> >> > Anyone have thoughts on this?
> >> > My votes would be B, and if that is not possible A and C.
> >> >
> >> > Thanks,
> >> > Dan
> >> >
> >> > [1] https://issues.apache.org/jira/browse/NIFI-438
> >> > [2] https://maven.apache.org/surefire/maven-failsafe-plugin/
> >>
>
>
>
> --
> Sean
>

Re: Fulfilling a pull request

[dan bress]

Whoops, step 5 should be
 5) mvn clean install -Pcontrib-check

On Mon, Oct 12, 2015 at 10:00 AM dan bress <danbr...@gmail.com> wrote:

> Apache NiFi Committers,
>I am working on fulfilling a pull request(NIFI-774), and wanted to make
> sure I am doing it right.
>
>These are the steps that I plan on following
>
>1) git remote add yu https://github.com/yu-iskw/nifi.git
>2) git pull yu
>3) git checkout master
>4) git merge --no-ff yu/NIFI-774
>5) clean install install -Pcontrib-check
>6) double check code and inspect release
>7) git push origin
>
> How does that sound to you guys?
>
> Thanks,
> Dan
>

Re: Fulfilling a pull request

[dan bress]

Aldrin,
   Awesome, I appreciate your perspective on this.  I'll check out option
#2.  Thanks!

Dan

On Mon, Oct 12, 2015 at 12:42 PM Aldrin Piri <aldrinp...@gmail.com> wrote:

> That way seems reasonable and should work.  The only negative, and I use
> the term loosely, is the continuous addition of remotes as new contributors
> provide submissions (certainly a nice problem to have!).  There are a few
> other ways you can tackle this to combat adding a remote for each user that
> may come along, and I'll list two I've encountered as a quick reference:
>
> * Grabbing a patch file:  You are able to append .patch onto any pull
> request URL to get a patch file that you can apply in the fashion to which
> you are accustomed.  [1]
> * Getting PRs directly from Github via [2] or [3]
>
> I primarily use the method of [2] myself.  There are some caveats about how
> this functions in general listed in the comments, but for the case of
> working with NiFi and treating github as another remote, has been a pretty
> streamlined process for me.  [3] is mechanically quite similar to [2], but
> does a full path to the PR in the remote.
>
> [1] https://github.com/blog/967-github-secrets#diff-patch
> [2] https://gist.github.com/piscisaureus/3342247
> [3]
>
> https://help.github.com/articles/checking-out-pull-requests-locally/#modifying-an-inactive-pull-request-locally
>
> On Mon, Oct 12, 2015 at 10:36 AM, dan bress <danbr...@gmail.com> wrote:
>
> > Whoops, step 5 should be
> >  5) mvn clean install -Pcontrib-check
> >
> > On Mon, Oct 12, 2015 at 10:00 AM dan bress <danbr...@gmail.com> wrote:
> >
> > > Apache NiFi Committers,
> > >I am working on fulfilling a pull request(NIFI-774), and wanted to
> > make
> > > sure I am doing it right.
> > >
> > >These are the steps that I plan on following
> > >
> > >1) git remote add yu https://github.com/yu-iskw/nifi.git
> > >2) git pull yu
> > >3) git checkout master
> > >4) git merge --no-ff yu/NIFI-774
> > >5) clean install install -Pcontrib-check
> > >6) double check code and inspect release
> > >7) git push origin
> > >
> > > How does that sound to you guys?
> > >
> > > Thanks,
> > > Dan
> > >
> >
>

"Integration" style Unit Tests

[dan bress]

Devs,
   While working on integrating and testing the work Yu did for
NIFI-774/DeleteS3Object, I noticed that a few of the unit tests for the
processors in that AWS bundle(Put and Fetch S3) actually interact with S3
directly, and were marked as @Ignore.  If I wanted to un @Ignore them and
actually run them I needed to set up an AWS account, then copy the
credentials into an aws-credentials.properties file and put that in my home
directory to get the tests to pass.  This dashed my hopes of a relatively
simple merge and turned it into a bit of work for me.  I'm not blaming Yu
or anyone for this, just wanted to open up a discussion on better ways of
solving this.

Problem:
@Ignore'd tests don't get run, probably ever.  Why?  Because running them
is a pain in the butt, I agree with NIFI-438[1] "If tests could talk they
would say don't @Ignore me".  I appreciate that there are special
circumstances for using this, but it would probably benefit everyone if we
sure we used it only for the truly special circumstances.


Solutions:
a. Run these tests using the failsafe plugin[2] instead of the surefire
plugin.  This way they get run every time, and if they fail that
information gets reported but it does not stop the build.
b. Mock out the service(I appreciate that this may not always be possible)
c. Provide instructions somewhere so that someone with no experience with
these processors/tests can run them.

Anyone have thoughts on this?
My votes would be B, and if that is not possible A and C.

Thanks,
Dan

[1] https://issues.apache.org/jira/browse/NIFI-438
[2] https://maven.apache.org/surefire/maven-failsafe-plugin/

Re: How to delete docs for renamed processor

[Dan Bress]

Rick,
   Can you confirm whether the old processor name still shows up in the "new 
processor" UI when you try and drag a new processor on the graph?

   Also did you do a 'mvn clean install' on your nar, or just 'mvn install'?

Dan Bress
Software Engineer
ONYX Consulting Services


From: Rick Braddy <rbra...@softnas.com>
Sent: Wednesday, September 9, 2015 3:00 PM
To: dev@nifi.apache.org
Subject: RE: How to delete docs for renamed processor

Thanks Mark.  Unfortunately, that didn't work - it's still there.

Rick

-Original Message-
From: Mark Payne [mailto:marka...@hotmail.com]
Sent: Wednesday, September 09, 2015 1:45 PM
To: dev@nifi.apache.org
Subject: RE: How to delete docs for renamed processor

Rick,

You can try deleting the nifi/work directory and restarting.

Thanks
-Mark


> From: rbra...@softnas.com
> To: dev@nifi.apache.org
> Subject: How to delete docs for renamed processor
> Date: Wed, 9 Sep 2015 18:42:05 +
>
> Hi,
>
> Hopefully a quick answer. I have renamed a custom processor and I still see 
> the processor by the old name showing up in the user docs. Is there a way to 
> delete the docs related to the old processor name?
>
> Thanks
> Rick
>

Re: eliminate nifi-parent, split out nifi-nar-maven-plugin, have nifi in its own tree

[Dan Bress]

No feedback on this one in a few days.  I will email INFRA now asking for the 
github mirrors.

Dan Bress
Software Engineer
ONYX Consulting Services


From: Dan Bress
Sent: Wednesday, August 19, 2015 12:33 PM
To: dev@nifi.apache.org
Subject: Re: eliminate nifi-parent, split out nifi-nar-maven-plugin, have nifi 
in its own tree

  Will not be asking to have them mirrored to Github as it
  doesn't seem worth it/necessary.

I'm going to suggest that we do have Github mirrors for both of these projects. 
 I find Github makes it very easy to navigate the repos as well as pretty 
formatting the nifi-site/Readme.md which provides instructions for how to 
deploy the site.

Does anyone object to me asking INFRA to provide github mirrors for nifi-site 
and nifi-maven?

Dan Bress
Software Engineer
ONYX Consulting Services


From: Joe Witt joe.w...@gmail.com
Sent: Saturday, August 15, 2015 1:45 PM
To: dev@nifi.apache.org
Subject: Re: eliminate nifi-parent, split out nifi-nar-maven-plugin, have nifi 
in its own tree

All,

This effort as tracked under NIFI-850 is completed.  The infra tickets
were handled in like 5 minutes (woot infra!).  Today I moved
nifi-nar-maven-plugin folder contents into the repo of nifi-maven and
moved nifi-site folder contents into nifi-site repo.  After that the
nifi-parent items were denormalized out into nifi/pom.xml and
nifi-maven/pom.xml as appropriate.  Readme files updated, website
updated, all pushed as appropriate.

Now when you pull down the latest code (currently on develop) you can
immediately build without needing to change directories or build other
things first.

That closes out this thread.  Next step will be to complete NIFI-857
which will result in the termination of the develop branch altogether.
But i'll wait and make sure that thread stays quiet the rest of the
day.

Thanks
Joe

On Thu, Aug 13, 2015 at 3:01 PM, Tony Kurc trk...@gmail.com wrote:
 Ryan,
 Having develop and master was due to the influence of git flow [1].

 [1] http://nvie.com/posts/a-successful-git-branching-model/

 On Thu, Aug 13, 2015 at 2:11 PM, Joey Echeverria joe...@gmail.com wrote:

 Currently master is the same as the last release tag.

 On Thu, Aug 13, 2015 at 1:51 PM, Ryan Blue b...@cloudera.com wrote:
  What is the current distinction between master and develop? Master is
 stable
  and develop is where new changes go? The reason I suggest just having
 master
  is that it follows the convention that other projects use. Master is
 where
  new development happens and releases or more stable branches are marked
  appropriately.
 
  rb
 
 
  On 08/13/2015 08:46 AM, Joe Witt wrote:
 
  All,
 
  Am filing the infra tickets now.  I forgot that we had 'nifi-site' at
  the root level too.  So requesting two new git repositories in Apache
  Infra.  Will not be asking to have them mirrored to Github as it
  doesn't seem worth it/necessary.
 
  'nifi-maven'   https://issues.apache.org/jira/browse/INFRA-10119
  'nifi-site'   https://issues.apache.org/jira/browse/INFRA-10120
 
  Actions:
  Once these two new git repositories are created i will move the
  appropriate nifi-nar-maven-plugin items into it and terminate the
  current directory.  Then I'll move the nifi-site directory content
  into the new nifi-site repository and then delete the directory.
 
  Once that is sorted we can discuss whether we care to keep
  develop/master or simply go to master as Ryan suggests.
 
  Thanks
  Joe
 
  On Mon, Aug 10, 2015 at 5:13 PM, Joe Witt joe.w...@gmail.com wrote:
 
  Ryan
 
  Correct the latest code depends on latest nifi nar maven plugin.
 
  I would be absolutely fine personally with eliminating develop and just
  using master.  Given that the releases are tagged i personally dont get
  the
  value here vs the extra work required.
 
  Anybody feel strongly for keeping master and dev as they are and if so
  can
  you please state how the current model has helped you contribute or how
  the
  proposed model would not?
 
  Thanks
  Joe
 
  On Aug 10, 2015 11:43 AM, Ryan Blue b...@cloudera.com wrote:
 
 
  +1
 
  I think separate git repos is a great idea. One thing to clarify, too:
  most of the time the nifi project relies on the last
  nifi-nar-maven-plugin
  release, right? So that should be transparent for most people building
  the
  project. It would only be awkward for someone updating the maven
 plugin
  and
  testing it out locally because the develop branch should always track
 a
  release.
 
  Speaking of the develop branch... what about using master like most
  projects after this change?
 
  rb
 
  On 08/10/2015 07:32 AM, Joe Witt wrote:
 
 
  Team,
 
  We've seen and heard the confusion of folks trying to build NiFi's
  goofy three step build process with parent, nar plugin, and nifi.  I
  propose to do the following:
 
  1) Eliminate the nifi-parent by pushing anything necessary back into
  nifi-nar-maven-plugin.  The DRY

Re: apache nifi oscon briefing

[Dan Bress]

+1

Dan Bress
Software Engineer
ONYX Consulting Services


From: Corey Flowers cflow...@onyxpoint.com
Sent: Friday, August 21, 2015 7:28 AM
To: dev@nifi.apache.org
Subject: Re: apache nifi oscon briefing

+1

Sent from my iPhone

 On Aug 21, 2015, at 6:53 AM, estrada.a...@gmail.com 
 estrada.a...@gmail.com wrote:

 +1

 Sent from my iPhone

 On Aug 21, 2015, at 5:49 AM, Jennifer Barnabee jennifer.barna...@gmail.com 
 wrote:

 Fine by me. It would be a good addition.

 Sent from my iPhone

 On Aug 20, 2015, at 9:04 PM, Joe Witt joe.w...@gmail.com wrote:

 Team,

 Does anyone object to me adding this briefing to our apache nifi webcasts 
 page?

 https://www.youtube.com/watch?v=sQCgtCoZyFQ

 It is the talk I gave at OSCON this July on Apache NiFi and its
 relationship to messaging and dataflow.

 After a couple days if all seems fine I'll go ahead and update the
 site to point to it.

 Thanks
 Joe

Re: Lots o branches in git

[Dan Bress]

My stale branches have been deleted.

Dan Bress
Software Engineer
ONYX Consulting Services


From: Mike Drob mad...@cloudera.com
Sent: Monday, August 17, 2015 10:07 AM
To: dev@nifi.apache.org
Subject: Re: Lots o branches in git

Also useful is
git remote prune origin

which will clean up your local tracking branches for remote branches that
have been deleted. You can specify different remote names instead of
origin, as well.


On Mon, Aug 17, 2015 at 7:44 AM, Joe Witt joe.w...@gmail.com wrote:

 Thanks Dan.

 Please be sure you did the command needed to remove the remote/origin
 branch.  It is possible that the email just hasn't come through but i
 suspect not as the ASF git repo still shows them.  This is the command
 set you want:

 #remove remote branch
 git push origin —delete NIFI-633
 git push origin —delete NIFI-632
 git push origin —delete NIFI-680
 git push origin —delete NIFI-682
 git push origin —delete NIFI-704
 git push origin —delete NIFI-735


 Thanks
 Joe

 On Mon, Aug 17, 2015 at 8:32 AM, Dan Bress dbr...@onyxconsults.com
 wrote:
  Joe, thanks for the heads up.  I'll delete all of mine, as they are all
 resolved in JIRA
  NIFI-633: danbress
  NIFI-632: danbress
  NIFI-680: danbress
  NIFI-682: danbress
  NIFI-704: danbress
  NIFI-735: danbress
 
  Dan Bress
  Software Engineer
  ONYX Consulting Services
 
  
  From: Joe Witt joew...@apache.org
  Sent: Sunday, August 16, 2015 9:34 PM
  To: dev@nifi.apache.org
  Subject: Lots o branches in git
 
  Team,
 
  Friendly reminder regarding branches that appear stale.  If you no
  longer need them please consider removing them.
 
  Last activity more than 2 months ago:
  --
  ListHDFS : markup
  NIFI-25: markap
  NIFI-85: mcgilma
  NIFI-180: mcgilma
  NIFI-250: mcgilma
  NIFI-271: apiri
  NIFI-292: mcgilma
  NIFI-353: mcgilma
  NIFI-376: mcgilma
  NIFI-433: devriesb
  NIFI-475: mcgilma
  NIFI-521: mcgilma
  NIFI-633: danbress
  improve-pro-performance: markap, mcgilma
  journaling-prov-repo: markap
  prov-query-language: markap
 
  Seems definitely eligible to remove (release is done and tagged):
  --
  release-nifi-0.1.0-rc13: markup
  release-nifi-0.2.0-incubating: apiri
  release-nifi-0.2.0-incubating-rc1: mcgilma
 
  Active within the last 2 months:
  ---
  NIFI-632: danbress
  NIFI-640: mcgilma
  NIFI-680: danbress
  NIFI-682: danbress
  NIFI-704: danbress
  NIFI-724: mcgilma
  NIFI-731: markap
  NIFI-735: danbress
  NIFI-744: markap
  NIFI-793: markap
  NIFI-818: markap
 
  Reminder on commands that can help:
  ———
  #remove local branch
  git branch -D branch name
  #remove remote branch
  git push origin —delete branch name
 
  Thanks
  Joe

Re: Process to create multiple files

[Dan Bress]

plj,
   I would not recommend having this processor create multiple thumbnails.  
What I would recommend is the following:

   Create a new processor called CreateThumbnail or RescaleImage

Then have a configuration on the processor that says what size the output 
image should be(e.g. 128x128, or 1/X of original size).

Your new processor will read in the incoming image, and rescale it down to 
the user specified size and pass it forward.

 Now if you want to create a 128x128 64x64 and 32x32 sized images you would 
do the following.

(GetFile)-(RescaleImage configured to 128x128)-(PutFile)
  |-(RescaleImage configured to 64x64)-(PutFile) 
  \-(RescaleImage configured to 32x32)-(PutFile)

Where GetFile has 3 success relationships, each going to a different 
RescaleImage processor.

I think it makes more sense to have one processor create one file, then you can 
use the flow to visually configure how many copies of the file you want to 
make.  This should make this processor simpler and more reusable.
 

Dan Bress
Software Engineer
ONYX Consulting Services


From: plj p...@mitre.org
Sent: Monday, August 17, 2015 1:27 PM
To: d...@nifi.incubator.apache.org
Subject: Process to create  multiple files

Howdy,

  I'm new to NiFi so please bear with me.  What I want to accomplish is:
  read an image file
 process the file to create one or more thumbnails from the image.
 Send the resulting thumbnails along the flow

So I can use GetFile to read the file and then send it along.  I think I
need to write a custom java processor that will process the image file and
then send each of the thumbnail files (say .jpg for now) on to the next
thing in the flow (say PutFile for example).

Are there suggestions on what I should implement or extend to create my
custom processor?  It will take in one file and output multiple files.
Would extending PutFile so that it processed and then puts each thumbnail
on the flow be a good strategy?  Other ideas?

Thank you,





--
View this message in context: 
http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Process-to-create-multiple-files-tp2510.html
Sent from the Apache NiFi (incubating) Developer List mailing list archive at 
Nabble.com.

Re: [DISCUSS] Removal of the 'master' vs 'develop' distinction

[Dan Bress]

+0.  Our default branch is set to 'develop', so when you clone apache-nifi from 
git, you are automatically looking at the 'develop' branch, right?  To me, this 
is a straight forward indicator of where I should be working.

I thought we set this up a little while ago to avoid the confusion?

Dan Bress
Software Engineer
ONYX Consulting Services


From: Ryan Blue b...@cloudera.com
Sent: Thursday, August 13, 2015 4:04 PM
To: dev@nifi.apache.org
Subject: Re: [DISCUSS] Removal of the 'master' vs 'develop' distinction

+1 to removing the distinction. Master is the default branch in a lot of
projects and I would argue that is the common expectation. It sounds
like we can do gitflow without a separate develop branch (or at least it
isn't too painful) so doing what new people tend to expect is a good thing.

rb

On 08/13/2015 12:55 PM, Mark Payne wrote:
 I think the issue here is less about gitflow being hard and more about it 
 being confusing.
 We have had numerous people write to the dev list about why the thing that 
 they checked out
 doesn't have what they expect.

 Even being very experience with NiFi, I've cloned the repo a couple of times 
 to new VM's
 and forgotten to checkout develop before proceeding.

 I think that gitflow has its merits, but like any other avenue we go down, 
 it's important to weigh
 pros against cons. Frankly, I think that anything that leads to confusion for 
 newcomers (thereby
 discouraging community growth) had better have some very strong benefits.

 That being said, I don't personally see a lot of benefit in this environment, 
 so I would
 be a +1 to remove the distinction between the two branches.


 
 Date: Thu, 13 Aug 2015 15:45:00 -0400
 Subject: Re: [DISCUSS] Removal of the 'master' vs 'develop' distinction
 From: a...@adamtaft.com
 To: dev@nifi.apache.org

 The difficulties of using the gitflow workflow and the release process can
 be significantly reduced with good tooling. I'm currently using the
 jgit-flow [1][2] maven plugin with very good success. It handles and
 manages feature, release, and hotfix branches seemlessly. And it avoids
 common problems with the normal maven release plugin for gitflow.

 Before abandoning gitflow, the community should seriously consider tooling
 that makes it more usable. I'm not going to argue the merits of gitlab
 flow or any other workflows. But clearly, abandoning gitflow because it's
 hard is likely the wrong driver, if tooling exists to make it better.

 [1]
 http://blogs.atlassian.com/2013/05/maven-git-flow-plugin-for-better-releases/

 [2] https://bitbucket.org/atlassian/jgit-flow/wiki/Home




 On Thu, Aug 13, 2015 at 2:58 PM, Bryan Bende bbe...@gmail.com wrote:

 If we worked on master and had a prod branch that was the last release,
 then we have the same thing we do now, just with different names. This
 would be GitLab Flow as Brandon pointed out.

 That being said, I don't have experience with the release process, and
 maybe the prod branch does not provide any value for us. The prod branch
 would normally be used to create quick fix branches based off production,
 or when doing automated/continuous deployments to a production system, but
 if we aren't doing either of those things then maybe it is not worth it.

 -Bryan

 On Thu, Aug 13, 2015 at 2:23 PM, Brandon DeVries b...@jhu.edu wrote:

 Personally, I still think GitLab Flow[1] is all we need for us to be
 Really
 Useful Engines.

 [1] https://about.gitlab.com/2014/09/29/gitlab-flow/

 Brandon

 On Thu, Aug 13, 2015 at 2:15 PM Joe Witt joew...@apache.org wrote:

 Resending
 On Aug 13, 2015 12:22 PM, Joe Witt joe.w...@gmail.com wrote:

 Team,

 It was proposed by Ryan Blue on another thread that we consider
 dropping the master vs develop distinction. In the interest of his,
 in my view, very good point I didn't want it to get buried in that
 thread.

 [1] is the thread when we last discussed gitflow/develop/master on
 entry to the incubator.

 And from that thread here is the part I wish I had better understood
 when the wise Mr Benson said it:

 Another issue with gitflow is the master branch. The master branch
 is
 supposed to get merged to for releases. The maven-release-plugin
 won't
 do that, and the jgitflow plugin is unsafe. So one option is to 'use
 gitflow' but not bother with the master versus develop distinction,
 the other is to do manual merges to master at release points.

 I think we should follow this guidance: 'use gitflow' but not bother
 with the master versus develop distinction. I say this from having
 done the release management job now a couple of times including
 having
 done a 'hotfix'.

 My comments here are not a rejection of that master/develop concept
 in
 general. It is simply pointing out that for the Apache NiFi
 community
 it is not adding value but is creating confusion and delay [2].

 Thanks
 Joe

 [1] http://s.apache.org/GIW
 [2] Sir Topham Hatt