Re: Web site fixes/corrections

[Dan Bress]

If you are still working on this Aldrin, I noticed that Benson's name is 
spelled wrong on the people page [1] 
bimargulies Benson Margulie 

[1]http://nifi.apache.org/people.html

Dan Bress
Software Engineer
ONYX Consulting Services


From: Joe Witt 
Sent: Tuesday, July 21, 2015 6:02 PM
To: dev@nifi.apache.org
Subject: Re: Web site fixes/corrections

I think Sean's proposal is a good balance.  We want it to be easy for
folks to contribute to making the website awesome-er.  We have a good
start but we have a long way to go.  For simple cleanup/maintenance
things a single cover-all JIRA should suffice.  If someone wants to
work a new concept/page/theme then that should have its own ticket.

On Tue, Jul 21, 2015 at 3:00 PM, Sean Busbey  wrote:
> I think a single "cleanup" jira that covers everything fixed in one go
> would be good, but I know that I tend to fall on the jira-for-everything
> side of things.
>
> On Tue, Jul 21, 2015 at 4:56 PM, Aldrin Piri  wrote:
>
>> All,
>>
>> I have made fixes to the website inclusive of items such as correcting
>> broken links, typos, and the like and have never created issues for these
>> fixes as I viewed them to be minor changes.
>>
>> I was curious if this practice seemed acceptable or if the community
>> preferred an issue for each such change.
>>
>> Thanks!
>>
>
>
>
> --
> Sean

Re: Write-Ahead-Log package name change?

[Dan Bress]

I'm fine with the package name being changed in 0.3.0

Dan Bress
Software Engineer
ONYX Consulting Services


From: Mark Payne 
Sent: Wednesday, August 5, 2015 3:01 PM
To: dev@nifi.apache.org
Subject: RE: Write-Ahead-Log package name change?

Ryan,

The WAL is certainly not defined in the nifi-api. But it does live in the 
nifi-commons module. Not entirely sure if i would consider it "public" or not.

My suggestion is to change the package name for the 0.3.0 release, which is a 
minor version.

Thanks
-Mark


> Date: Wed, 5 Aug 2015 11:38:02 -0700
> From: b...@cloudera.com
> To: dev@nifi.apache.org
> Subject: Re: Write-Ahead-Log package name change?
>
> Is the WAL a public API? I thought that it was internal, in which case a
> rename should be fine. Otherwise we would have to bump the major version
> number (or minor depending on discussion) to account for the change.
>
> rb
>
> On 08/03/2015 11:53 AM, Mark Payne wrote:
>> Hello,
>>
>> I recently realized that the nifi-write-ahead-log module (under 
>> nifi-commons) is using a package name of "org.wali" instead of 
>> "org.apache.nifi.wal"
>>
>> This has been the package name since the software was open sourced, 
>> unfortunately. I would like to change the package name for the 0.3.0 version 
>> of NiFi, if there are no objections.
>>
>> The pre-0.3.0 versions would, of course, still be available if anyone has a 
>> dependency on the classes, but I would like to get this fixed so that it is 
>> correct going forward.
>>
>> Is there any reason that we cannot change this for the 0.3.0 release?
>>
>> Thanks
>> -Mark
>>
>
>
> --
> Ryan Blue
> Software Engineer
> Cloudera, Inc.

Re: [DISCUSS] Feature proposal: Read-only mode as default

[Dan Bress]

+1 to exactly what Mark described in his last email for a system wide 
preference.

Although I'm curious how you leave read-only mode and get into edit mode?  And 
how do you leave edit mode and go back to read-only mode?

On one hand, if I do not intend to edit the graph and I accidentally move a 
processor I probably don't want it to prompt me "do you want to enter edit 
mode?"
-In read-only mode, I think it would be a nice user experience to click 
anywhere on the graph(including on a processor) and it moves the entire graph.  
On the other hand, if I right click a processor and hit configure I'd like to 
leave read-only mode and go into edit mode.  I'm not sure I'd even want to be 
prompted with "do you want to enter edit mode?" here since I obviously do.


Dan Bress
Software Engineer
ONYX Consulting Services


From: Mark Payne 
Sent: Monday, August 10, 2015 3:43 PM
To: dev@nifi.apache.org
Subject: RE: [DISCUSS] Feature proposal: Read-only mode as default

I'm definitely a +1. I accidentally drag processors all the time when I'm 
panning around a large graph.

I can understand how someone would get annoyed with this, though, and I can 
also appreciate the desire
to not start storing user preferences. However, I think we should probably at 
least supply a system-level
configuration for whether or not to have "read-only" the default mode. Then 
administrators can turn it on
or off, depending on the users of the system.




> Date: Sat, 8 Aug 2015 20:50:07 -0400
> Subject: Re: [DISCUSS] Feature proposal: Read-only mode as default
> From: a...@adamtaft.com
> To: dev@nifi.apache.org
>
> Thinking about it some more, I don't mind the concept of "read only"
> toggle. Maybe it's not set by default, but it could be a really easy UI
> element to add somewhere. Just a little slider-toggle element. [1]
>
> In theory, this might be a UI only feature, it wouldn't strictly need
> support in the backend api (just guessing). The logic is seemingly already
> there, similar user experience as non-DFMs.
>
> Anyway, +1 to the concept of read-only toggle mode. -1 for making it
> default, but the user interface element should be easy to work either way.
>
> Also agree that undo support might be free if versioning is added.
>
> [1] http://turbo.premiumpixels.com/wp-content/uploads/2011/04/preview2.jpg
>
>
>
>
> On Sat, Aug 8, 2015 at 3:05 PM, Joe Witt  wrote:
>
>> Ryan - the other useful case for read-only is basically when is
>> scanning around the graph and accidentally moves a processor or
>> relationship. By no means a big deal. The idea here was to make it
>> explicit though that the user wishes to go into an edit mode.
>>
>> I do think the undo mechanism plays well and you're right that we can
>> just focus on tightening up the delete case.
>>
>> Sounds like the prevailing view is to avoid read-only as a mode but
>> rather to make it more explicit whenever we delete - and potentially
>> move we could make more specific rather than simply them having
>> clicked and dragged which is ambiguous with the process of panning.
>>
>> On Sat, Aug 8, 2015 at 2:57 PM, Ryan Blue  wrote:
>>> I'm not a big fan of having a read-only mode by default. It sounds like
>>> something that would be frustrating for users when they try to make
>> changes
>>> and then have to figure out how to switch modes.
>>>
>>> I think a clearer picture of the problem we're trying to solve would
>> help my
>>> understanding. I'm primarily thinking of the delete case you mentioned
>> with
>>> these comments...
>>>
>>> If we're talking about accidentally deleting processors, then the current
>>> mechanisms (IIRC) work pretty well: not deleting a running processor, one
>>> that has live incoming connections, etc. If those rules are
>> insufficient, I
>>> would explore extending them rather than having a global read-only mode.
>>>
>>> For the case where the wrong processor is selected because it is off the
>>> screen, maybe having the confirmation pop up if anything affected wasn't
>>> displayed to confirm? That way we don't have confirmations all the time
>> but
>>> still don't do unexpected things.
>>>
>>> I really like the idea of "undo" as well. If that is limited to
>> processors
>>> that weren't running (because you can't delete ones that are), then that
>>> makes the undo operation easier to implement.
>>>
>>> rb
>>>
>>

Re: [DISCUSS] Removal of the 'master' vs 'develop' distinction

[Dan Bress]

+0.  Our default branch is set to 'develop', so when you clone apache-nifi from 
git, you are automatically looking at the 'develop' branch, right?  To me, this 
is a straight forward indicator of where I should be working.

I thought we set this up a little while ago to avoid the confusion?

Dan Bress
Software Engineer
ONYX Consulting Services


From: Ryan Blue 
Sent: Thursday, August 13, 2015 4:04 PM
To: dev@nifi.apache.org
Subject: Re: [DISCUSS] Removal of the 'master' vs 'develop' distinction

+1 to removing the distinction. Master is the default branch in a lot of
projects and I would argue that is the common expectation. It sounds
like we can do gitflow without a separate develop branch (or at least it
isn't too painful) so doing what new people tend to expect is a good thing.

rb

On 08/13/2015 12:55 PM, Mark Payne wrote:
> I think the issue here is less about gitflow being "hard" and more about it 
> being confusing.
> We have had numerous people write to the dev list about why the thing that 
> they checked out
> doesn't have what they expect.
>
> Even being very experience with NiFi, I've cloned the repo a couple of times 
> to new VM's
> and forgotten to checkout develop before proceeding.
>
> I think that gitflow has its merits, but like any other avenue we go down, 
> it's important to weigh
> pros against cons. Frankly, I think that anything that leads to confusion for 
> newcomers (thereby
> discouraging community growth) had better have some very strong benefits.
>
> That being said, I don't personally see a lot of benefit in this environment, 
> so I would
> be a +1 to remove the distinction between the two branches.
>
>
> 
>> Date: Thu, 13 Aug 2015 15:45:00 -0400
>> Subject: Re: [DISCUSS] Removal of the 'master' vs 'develop' distinction
>> From: a...@adamtaft.com
>> To: dev@nifi.apache.org
>>
>> The difficulties of using the gitflow workflow and the release process can
>> be significantly reduced with good tooling. I'm currently using the
>> jgit-flow [1][2] maven plugin with very good success. It handles and
>> manages feature, release, and hotfix branches seemlessly. And it avoids
>> common problems with the normal maven release plugin for gitflow.
>>
>> Before abandoning gitflow, the community should seriously consider tooling
>> that makes it more usable. I'm not going to argue the merits of gitlab
>> flow or any other workflows. But clearly, abandoning gitflow because it's
>> "hard" is likely the wrong driver, if tooling exists to make it better.
>>
>> [1]
>> http://blogs.atlassian.com/2013/05/maven-git-flow-plugin-for-better-releases/
>>
>> [2] https://bitbucket.org/atlassian/jgit-flow/wiki/Home
>>
>>
>>
>>
>> On Thu, Aug 13, 2015 at 2:58 PM, Bryan Bende  wrote:
>>
>>> If we worked on master and had a prod branch that was the last release,
>>> then we have the same thing we do now, just with different names. This
>>> would be GitLab Flow as Brandon pointed out.
>>>
>>> That being said, I don't have experience with the release process, and
>>> maybe the prod branch does not provide any value for us. The prod branch
>>> would normally be used to create quick fix branches based off production,
>>> or when doing automated/continuous deployments to a production system, but
>>> if we aren't doing either of those things then maybe it is not worth it.
>>>
>>> -Bryan
>>>
>>> On Thu, Aug 13, 2015 at 2:23 PM, Brandon DeVries  wrote:
>>>
>>>> Personally, I still think GitLab Flow[1] is all we need for us to be
>>> Really
>>>> Useful Engines.
>>>>
>>>> [1] https://about.gitlab.com/2014/09/29/gitlab-flow/
>>>>
>>>> Brandon
>>>>
>>>> On Thu, Aug 13, 2015 at 2:15 PM Joe Witt  wrote:
>>>>
>>>>> Resending
>>>>> On Aug 13, 2015 12:22 PM, "Joe Witt"  wrote:
>>>>>
>>>>>> Team,
>>>>>>
>>>>>> It was proposed by Ryan Blue on another thread that we consider
>>>>>> dropping the master vs develop distinction. In the interest of his,
>>>>>> in my view, very good point I didn't want it to get buried in that
>>>>>> thread.
>>>>>>
>>>>>> [1] is the thread when we last discussed gitflow/develop/master on
>>>>>> entry to the inc

Re: [DISCUSS] Removal of the 'master' vs 'develop' distinction

[Dan Bress]

Ah, I didn't realize that was a github only thing [1], I take-back my early 
comment and can now see how this is confusing.

[1] 
http://mail-archives.apache.org/mod_mbox/nifi-dev/201501.mbox/%3CCALhtWke141nTsCdA4tHnZXOJ1UGhtZurLwvDsjBxH_G=86n...@mail.gmail.com%3E

Dan Bress
Software Engineer
ONYX Consulting Services


From: Joe Witt 
Sent: Thursday, August 13, 2015 4:22 PM
To: dev@nifi.apache.org
Subject: Re: [DISCUSS] Removal of the 'master' vs 'develop' distinction

Nope.  That is just what is shown in github as the default.
On Aug 13, 2015 4:15 PM, "Dan Bress"  wrote:

> +0.  Our default branch is set to 'develop', so when you clone apache-nifi
> from git, you are automatically looking at the 'develop' branch, right?  To
> me, this is a straight forward indicator of where I should be working.
>
> I thought we set this up a little while ago to avoid the confusion?
>
> Dan Bress
> Software Engineer
> ONYX Consulting Services
>
> 
> From: Ryan Blue 
> Sent: Thursday, August 13, 2015 4:04 PM
> To: dev@nifi.apache.org
> Subject: Re: [DISCUSS] Removal of the 'master' vs 'develop' distinction
>
> +1 to removing the distinction. Master is the default branch in a lot of
> projects and I would argue that is the common expectation. It sounds
> like we can do gitflow without a separate develop branch (or at least it
> isn't too painful) so doing what new people tend to expect is a good thing.
>
> rb
>
> On 08/13/2015 12:55 PM, Mark Payne wrote:
> > I think the issue here is less about gitflow being "hard" and more about
> it being confusing.
> > We have had numerous people write to the dev list about why the thing
> that they checked out
> > doesn't have what they expect.
> >
> > Even being very experience with NiFi, I've cloned the repo a couple of
> times to new VM's
> > and forgotten to checkout develop before proceeding.
> >
> > I think that gitflow has its merits, but like any other avenue we go
> down, it's important to weigh
> > pros against cons. Frankly, I think that anything that leads to
> confusion for newcomers (thereby
> > discouraging community growth) had better have some very strong benefits.
> >
> > That being said, I don't personally see a lot of benefit in this
> environment, so I would
> > be a +1 to remove the distinction between the two branches.
> >
> >
> > 
> >> Date: Thu, 13 Aug 2015 15:45:00 -0400
> >> Subject: Re: [DISCUSS] Removal of the 'master' vs 'develop' distinction
> >> From: a...@adamtaft.com
> >> To: dev@nifi.apache.org
> >>
> >> The difficulties of using the gitflow workflow and the release process
> can
> >> be significantly reduced with good tooling. I'm currently using the
> >> jgit-flow [1][2] maven plugin with very good success. It handles and
> >> manages feature, release, and hotfix branches seemlessly. And it avoids
> >> common problems with the normal maven release plugin for gitflow.
> >>
> >> Before abandoning gitflow, the community should seriously consider
> tooling
> >> that makes it more usable. I'm not going to argue the merits of gitlab
> >> flow or any other workflows. But clearly, abandoning gitflow because
> it's
> >> "hard" is likely the wrong driver, if tooling exists to make it better.
> >>
> >> [1]
> >>
> http://blogs.atlassian.com/2013/05/maven-git-flow-plugin-for-better-releases/
> >>
> >> [2] https://bitbucket.org/atlassian/jgit-flow/wiki/Home
> >>
> >>
> >>
> >>
> >> On Thu, Aug 13, 2015 at 2:58 PM, Bryan Bende  wrote:
> >>
> >>> If we worked on master and had a prod branch that was the last release,
> >>> then we have the same thing we do now, just with different names. This
> >>> would be GitLab Flow as Brandon pointed out.
> >>>
> >>> That being said, I don't have experience with the release process, and
> >>> maybe the prod branch does not provide any value for us. The prod
> branch
> >>> would normally be used to create quick fix branches based off
> production,
> >>> or when doing automated/continuous deployments to a production system,
> but
> >>> if we aren't doing either of those things then maybe it is not worth
> it.
> >>>
> >>> -Bryan
> >>>
> >>> On Thu, Aug 13, 

Re: Lots o branches in git

[Dan Bress]

Joe, thanks for the heads up.  I'll delete all of mine, as they are all 
resolved in JIRA
NIFI-633: danbress
NIFI-632: danbress
NIFI-680: danbress
NIFI-682: danbress
NIFI-704: danbress
NIFI-735: danbress

Dan Bress
Software Engineer
ONYX Consulting Services


From: Joe Witt 
Sent: Sunday, August 16, 2015 9:34 PM
To: dev@nifi.apache.org
Subject: Lots o branches in git

Team,

Friendly reminder regarding branches that appear stale.  If you no
longer need them please consider removing them.

Last activity more than 2 months ago:
--
ListHDFS : markup
NIFI-25: markap
NIFI-85: mcgilma
NIFI-180: mcgilma
NIFI-250: mcgilma
NIFI-271: apiri
NIFI-292: mcgilma
NIFI-353: mcgilma
NIFI-376: mcgilma
NIFI-433: devriesb
NIFI-475: mcgilma
NIFI-521: mcgilma
NIFI-633: danbress
improve-pro-performance: markap, mcgilma
journaling-prov-repo: markap
prov-query-language: markap

Seems definitely eligible to remove (release is done and tagged):
--
release-nifi-0.1.0-rc13: markup
release-nifi-0.2.0-incubating: apiri
release-nifi-0.2.0-incubating-rc1: mcgilma

Active within the last 2 months:
---
NIFI-632: danbress
NIFI-640: mcgilma
NIFI-680: danbress
NIFI-682: danbress
NIFI-704: danbress
NIFI-724: mcgilma
NIFI-731: markap
NIFI-735: danbress
NIFI-744: markap
NIFI-793: markap
NIFI-818: markap

Reminder on commands that can help:
———
#remove local branch
git branch -D 
#remove remote branch
git push origin —delete 

Thanks
Joe

Re: Process to create multiple files

[Dan Bress]

plj,
   I would not recommend having this processor create multiple thumbnails.  
What I would recommend is the following:

   Create a new processor called "CreateThumbnail" or "RescaleImage"

Then have a configuration on the processor that says what size the output 
image should be(e.g. 128x128, or 1/X of original size).

Your new processor will read in the incoming image, and rescale it down to 
the user specified size and pass it forward.

 Now if you want to create a 128x128 64x64 and 32x32 sized images you would 
do the following.

(GetFile)->(RescaleImage configured to 128x128)->(PutFile)
  |->(RescaleImage configured to 64x64)->(PutFile) 
  \->(RescaleImage configured to 32x32)->(PutFile)

Where GetFile has 3 success relationships, each going to a different 
RescaleImage processor.

I think it makes more sense to have one processor create one file, then you can 
use the flow to visually configure how many copies of the file you want to 
make.  This should make this processor simpler and more reusable.
 

Dan Bress
Software Engineer
ONYX Consulting Services


From: plj 
Sent: Monday, August 17, 2015 1:27 PM
To: d...@nifi.incubator.apache.org
Subject: Process to create  multiple files

Howdy,

  I'm new to NiFi so please bear with me.  What I want to accomplish is:
  read an image file
 process the file to create one or more thumbnails from the image.
 Send the resulting thumbnails along the flow

So I can use GetFile to read the file and then send it along.  I think I
need to write a custom java processor that will process the image file and
then send each of the thumbnail files (say .jpg for now) on to the next
thing in the flow (say PutFile for example).

Are there suggestions on what I should implement or extend to create my
custom processor?  It will take in one file and output multiple files.
Would extending "PutFile" so that it processed and then puts each thumbnail
on the flow be a good strategy?  Other ideas?

Thank you,





--
View this message in context: 
http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Process-to-create-multiple-files-tp2510.html
Sent from the Apache NiFi (incubating) Developer List mailing list archive at 
Nabble.com.

Re: Lots o branches in git

[Dan Bress]

My stale branches have been deleted.

Dan Bress
Software Engineer
ONYX Consulting Services


From: Mike Drob 
Sent: Monday, August 17, 2015 10:07 AM
To: dev@nifi.apache.org
Subject: Re: Lots o branches in git

Also useful is
git remote prune origin

which will clean up your local tracking branches for remote branches that
have been deleted. You can specify different remote names instead of
origin, as well.


On Mon, Aug 17, 2015 at 7:44 AM, Joe Witt  wrote:

> Thanks Dan.
>
> Please be sure you did the command needed to remove the remote/origin
> branch.  It is possible that the email just hasn't come through but i
> suspect not as the ASF git repo still shows them.  This is the command
> set you want:
>
> #remove remote branch
> git push origin —delete NIFI-633
> git push origin —delete NIFI-632
> git push origin —delete NIFI-680
> git push origin —delete NIFI-682
> git push origin —delete NIFI-704
> git push origin —delete NIFI-735
>
>
> Thanks
> Joe
>
> On Mon, Aug 17, 2015 at 8:32 AM, Dan Bress 
> wrote:
> > Joe, thanks for the heads up.  I'll delete all of mine, as they are all
> resolved in JIRA
> > NIFI-633: danbress
> > NIFI-632: danbress
> > NIFI-680: danbress
> > NIFI-682: danbress
> > NIFI-704: danbress
> > NIFI-735: danbress
> >
> > Dan Bress
> > Software Engineer
> > ONYX Consulting Services
> >
> > 
> > From: Joe Witt 
> > Sent: Sunday, August 16, 2015 9:34 PM
> > To: dev@nifi.apache.org
> > Subject: Lots o branches in git
> >
> > Team,
> >
> > Friendly reminder regarding branches that appear stale.  If you no
> > longer need them please consider removing them.
> >
> > Last activity more than 2 months ago:
> > --
> > ListHDFS : markup
> > NIFI-25: markap
> > NIFI-85: mcgilma
> > NIFI-180: mcgilma
> > NIFI-250: mcgilma
> > NIFI-271: apiri
> > NIFI-292: mcgilma
> > NIFI-353: mcgilma
> > NIFI-376: mcgilma
> > NIFI-433: devriesb
> > NIFI-475: mcgilma
> > NIFI-521: mcgilma
> > NIFI-633: danbress
> > improve-pro-performance: markap, mcgilma
> > journaling-prov-repo: markap
> > prov-query-language: markap
> >
> > Seems definitely eligible to remove (release is done and tagged):
> > --
> > release-nifi-0.1.0-rc13: markup
> > release-nifi-0.2.0-incubating: apiri
> > release-nifi-0.2.0-incubating-rc1: mcgilma
> >
> > Active within the last 2 months:
> > ---
> > NIFI-632: danbress
> > NIFI-640: mcgilma
> > NIFI-680: danbress
> > NIFI-682: danbress
> > NIFI-704: danbress
> > NIFI-724: mcgilma
> > NIFI-731: markap
> > NIFI-735: danbress
> > NIFI-744: markap
> > NIFI-793: markap
> > NIFI-818: markap
> >
> > Reminder on commands that can help:
> > ———
> > #remove local branch
> > git branch -D 
> > #remove remote branch
> > git push origin —delete 
> >
> > Thanks
> > Joe
>

Help with new git repos

[Dan Bress]

Now that we have 3 git repos(nifi, nifi-site, nifi-maven-plugin), I think we 
could do a few things to make navigating them better.


1) Is this [1] page active anymore?  Has it been replaced by a non incubator 
version?  All if its references are to -incubator things... This is where I 
used to go to find links to our various repos.

2) Our primary website [2] doesn't have any links or references to the 
nifi-site repo, or the nifi-maven-plugin repo

3) The nifi-site repo has a nice Readme.md [3] that tells you how to make 
updates to the site... but since this isn't mirrored on github there's no way 
to view the formatted version


This left me confused when I was attempting to update the release guide last 
night for NIFI-851, because first I couldn't find the nifi-site repo, then I 
had a hard time reading the Readme.md. I think we could improve both of those 
things.


I don't know what the right thing to do about 1) is.  For 2) I think we should 
have links to those repos in the quickstart guide(or somewhere else) so people 
know they exist.  For 3) I think we should github mirror those repos.  I know 
Joe asked about this a few days ago and I didn't say anything because I didn't 
see the need at the time.  Now I do.  I think viewing the readme alone is worth 
it.  Plus I find github to be much betterfor navigating the repo and viewing 
logs.


Anyone have any thoughts on this?


[1] http://incubator.apache.org/projects/nifi.html

[2] http://nifi.apache.org

[3] 
https://git-wip-us.apache.org/repos/asf?p=nifi-site.git;a=blob;f=README.md;h=2441a3011cb5163bd440fb04da41b18c8ac21a68;hb=HEAD



Dan Bress
Software Engineer
ONYX Consulting Services

Re: Help with new git repos

[Dan Bress]

[1] fine with me
[2] My mistake, I was searching the page text for 'nifi-site' and 'nifi-maven' 
and didn't get any hits.  Upon actually reading the page with my eyeballs, I 
see them at the top of the page.  This works for me.
[3] I'll bring this up in the other thread, and if I get no negative feedback 
I'll email INFRA.

Thanks,

Dan Bress
Software Engineer
ONYX Consulting Services


From: Joe Witt 
Sent: Wednesday, August 19, 2015 11:49 AM
To: dev@nifi.apache.org
Cc: d...@nifi.incubator.apache.org
Subject: Re: Help with new git repos

[1] is dead.  It shall stay like that forever.

[2] It does.  See here: https://nifi.apache.org/quickstart.html
However, I agree we can make it more clear and if you had ideas there
would love to see em.

[3] Yeah we can ask the ASF infra folks to mirror the nifi-site and
nifi-maven repos to github.  Just didn't ask because seemed of minimal
value (to me).  But you alone wanting to is enough for me.  Please
feel free to file the INFRA request to have that done.  If you
need/want help on that just let me know.  Perhaps reply to that
previous email and see if anyone disagrees with that view.

Thanks
Joe

On Wed, Aug 19, 2015 at 11:43 AM, Dan Bress  wrote:
> Now that we have 3 git repos(nifi, nifi-site, nifi-maven-plugin), I think we 
> could do a few things to make navigating them better.
>
>
> 1) Is this [1] page active anymore?  Has it been replaced by a non incubator 
> version?  All if its references are to -incubator things... This is where I 
> used to go to find links to our various repos.
>
> 2) Our primary website [2] doesn't have any links or references to the 
> nifi-site repo, or the nifi-maven-plugin repo
>
> 3) The nifi-site repo has a nice Readme.md [3] that tells you how to make 
> updates to the site... but since this isn't mirrored on github there's no way 
> to view the formatted version
>
>
> This left me confused when I was attempting to update the release guide last 
> night for NIFI-851, because first I couldn't find the nifi-site repo, then I 
> had a hard time reading the Readme.md. I think we could improve both of those 
> things.
>
>
> I don't know what the right thing to do about 1) is.  For 2) I think we 
> should have links to those repos in the quickstart guide(or somewhere else) 
> so people know they exist.  For 3) I think we should github mirror those 
> repos.  I know Joe asked about this a few days ago and I didn't say anything 
> because I didn't see the need at the time.  Now I do.  I think viewing the 
> readme alone is worth it.  Plus I find github to be much betterfor navigating 
> the repo and viewing logs.
>
>
> Anyone have any thoughts on this?
>
>
> [1] http://incubator.apache.org/projects/nifi.html
>
> [2] http://nifi.apache.org
>
> [3] 
> https://git-wip-us.apache.org/repos/asf?p=nifi-site.git;a=blob;f=README.md;h=2441a3011cb5163bd440fb04da41b18c8ac21a68;hb=HEAD
>
>
>
> Dan Bress
> Software Engineer
> ONYX Consulting Services

Re: eliminate nifi-parent, split out nifi-nar-maven-plugin, have nifi in its own tree

[Dan Bress]

>> >> Will not be asking to have them mirrored to Github as it
>> >> doesn't seem worth it/necessary.

I'm going to suggest that we do have Github mirrors for both of these projects. 
 I find Github makes it very easy to navigate the repos as well as pretty 
formatting the nifi-site/Readme.md which provides instructions for how to 
deploy the site.

Does anyone object to me asking INFRA to provide github mirrors for nifi-site 
and nifi-maven?

Dan Bress
Software Engineer
ONYX Consulting Services


From: Joe Witt 
Sent: Saturday, August 15, 2015 1:45 PM
To: dev@nifi.apache.org
Subject: Re: eliminate nifi-parent, split out nifi-nar-maven-plugin, have nifi 
in its own tree

All,

This effort as tracked under NIFI-850 is completed.  The infra tickets
were handled in like 5 minutes (woot infra!).  Today I moved
nifi-nar-maven-plugin folder contents into the repo of nifi-maven and
moved nifi-site folder contents into nifi-site repo.  After that the
nifi-parent items were denormalized out into nifi/pom.xml and
nifi-maven/pom.xml as appropriate.  Readme files updated, website
updated, all pushed as appropriate.

Now when you pull down the latest code (currently on develop) you can
immediately build without needing to change directories or build other
things first.

That closes out this thread.  Next step will be to complete NIFI-857
which will result in the termination of the develop branch altogether.
But i'll wait and make sure that thread stays quiet the rest of the
day.

Thanks
Joe

On Thu, Aug 13, 2015 at 3:01 PM, Tony Kurc  wrote:
> Ryan,
> Having develop and master was due to the influence of git flow [1].
>
> [1] http://nvie.com/posts/a-successful-git-branching-model/
>
> On Thu, Aug 13, 2015 at 2:11 PM, Joey Echeverria  wrote:
>
>> Currently master is the same as the last release tag.
>>
>> On Thu, Aug 13, 2015 at 1:51 PM, Ryan Blue  wrote:
>> > What is the current distinction between master and develop? Master is
>> stable
>> > and develop is where new changes go? The reason I suggest just having
>> master
>> > is that it follows the convention that other projects use. Master is
>> where
>> > new development happens and releases or more stable branches are marked
>> > appropriately.
>> >
>> > rb
>> >
>> >
>> > On 08/13/2015 08:46 AM, Joe Witt wrote:
>> >>
>> >> All,
>> >>
>> >> Am filing the infra tickets now.  I forgot that we had 'nifi-site' at
>> >> the root level too.  So requesting two new git repositories in Apache
>> >> Infra.  Will not be asking to have them mirrored to Github as it
>> >> doesn't seem worth it/necessary.
>> >>
>> >> 'nifi-maven'   https://issues.apache.org/jira/browse/INFRA-10119
>> >> 'nifi-site'   https://issues.apache.org/jira/browse/INFRA-10120
>> >>
>> >> Actions:
>> >> Once these two new git repositories are created i will move the
>> >> appropriate nifi-nar-maven-plugin items into it and terminate the
>> >> current directory.  Then I'll move the nifi-site directory content
>> >> into the new nifi-site repository and then delete the directory.
>> >>
>> >> Once that is sorted we can discuss whether we care to keep
>> >> develop/master or simply go to master as Ryan suggests.
>> >>
>> >> Thanks
>> >> Joe
>> >>
>> >> On Mon, Aug 10, 2015 at 5:13 PM, Joe Witt  wrote:
>> >>>
>> >>> Ryan
>> >>>
>> >>> Correct the latest code depends on latest nifi nar maven plugin.
>> >>>
>> >>> I would be absolutely fine personally with eliminating develop and just
>> >>> using master.  Given that the releases are tagged i personally dont get
>> >>> the
>> >>> value here vs the extra work required.
>> >>>
>> >>> Anybody feel strongly for keeping master and dev as they are and if so
>> >>> can
>> >>> you please state how the current model has helped you contribute or how
>> >>> the
>> >>> proposed model would not?
>> >>>
>> >>> Thanks
>> >>> Joe
>> >>>
>> >>> On Aug 10, 2015 11:43 AM, "Ryan Blue"  wrote:
>> >>>>
>> >>>>
>> >>>> +1
>> >>>>
>> >>>> I think separate git repos is a great idea. One thing to clarify, too:
>> >>>> most of the ti

Re: apache nifi oscon briefing

[Dan Bress]

+1

Dan Bress
Software Engineer
ONYX Consulting Services


From: Corey Flowers 
Sent: Friday, August 21, 2015 7:28 AM
To: dev@nifi.apache.org
Subject: Re: apache nifi oscon briefing

+1

Sent from my iPhone

> On Aug 21, 2015, at 6:53 AM, "estrada.a...@gmail.com" 
>  wrote:
>
> +1
>
> Sent from my iPhone
>
>> On Aug 21, 2015, at 5:49 AM, Jennifer Barnabee  
>> wrote:
>>
>> Fine by me. It would be a good addition.
>>
>> Sent from my iPhone
>>
>>> On Aug 20, 2015, at 9:04 PM, Joe Witt  wrote:
>>>
>>> Team,
>>>
>>> Does anyone object to me adding this briefing to our apache nifi webcasts 
>>> page?
>>>
>>> https://www.youtube.com/watch?v=sQCgtCoZyFQ
>>>
>>> It is the talk I gave at OSCON this July on Apache NiFi and its
>>> relationship to messaging and dataflow.
>>>
>>> After a couple days if all seems fine I'll go ahead and update the
>>> site to point to it.
>>>
>>> Thanks
>>> Joe

Re: eliminate nifi-parent, split out nifi-nar-maven-plugin, have nifi in its own tree

[Dan Bress]

No feedback on this one in a few days.  I will email INFRA now asking for the 
github mirrors.

Dan Bress
Software Engineer
ONYX Consulting Services


From: Dan Bress
Sent: Wednesday, August 19, 2015 12:33 PM
To: dev@nifi.apache.org
Subject: Re: eliminate nifi-parent, split out nifi-nar-maven-plugin, have nifi 
in its own tree

>> >> Will not be asking to have them mirrored to Github as it
>> >> doesn't seem worth it/necessary.

I'm going to suggest that we do have Github mirrors for both of these projects. 
 I find Github makes it very easy to navigate the repos as well as pretty 
formatting the nifi-site/Readme.md which provides instructions for how to 
deploy the site.

Does anyone object to me asking INFRA to provide github mirrors for nifi-site 
and nifi-maven?

Dan Bress
Software Engineer
ONYX Consulting Services


From: Joe Witt 
Sent: Saturday, August 15, 2015 1:45 PM
To: dev@nifi.apache.org
Subject: Re: eliminate nifi-parent, split out nifi-nar-maven-plugin, have nifi 
in its own tree

All,

This effort as tracked under NIFI-850 is completed.  The infra tickets
were handled in like 5 minutes (woot infra!).  Today I moved
nifi-nar-maven-plugin folder contents into the repo of nifi-maven and
moved nifi-site folder contents into nifi-site repo.  After that the
nifi-parent items were denormalized out into nifi/pom.xml and
nifi-maven/pom.xml as appropriate.  Readme files updated, website
updated, all pushed as appropriate.

Now when you pull down the latest code (currently on develop) you can
immediately build without needing to change directories or build other
things first.

That closes out this thread.  Next step will be to complete NIFI-857
which will result in the termination of the develop branch altogether.
But i'll wait and make sure that thread stays quiet the rest of the
day.

Thanks
Joe

On Thu, Aug 13, 2015 at 3:01 PM, Tony Kurc  wrote:
> Ryan,
> Having develop and master was due to the influence of git flow [1].
>
> [1] http://nvie.com/posts/a-successful-git-branching-model/
>
> On Thu, Aug 13, 2015 at 2:11 PM, Joey Echeverria  wrote:
>
>> Currently master is the same as the last release tag.
>>
>> On Thu, Aug 13, 2015 at 1:51 PM, Ryan Blue  wrote:
>> > What is the current distinction between master and develop? Master is
>> stable
>> > and develop is where new changes go? The reason I suggest just having
>> master
>> > is that it follows the convention that other projects use. Master is
>> where
>> > new development happens and releases or more stable branches are marked
>> > appropriately.
>> >
>> > rb
>> >
>> >
>> > On 08/13/2015 08:46 AM, Joe Witt wrote:
>> >>
>> >> All,
>> >>
>> >> Am filing the infra tickets now.  I forgot that we had 'nifi-site' at
>> >> the root level too.  So requesting two new git repositories in Apache
>> >> Infra.  Will not be asking to have them mirrored to Github as it
>> >> doesn't seem worth it/necessary.
>> >>
>> >> 'nifi-maven'   https://issues.apache.org/jira/browse/INFRA-10119
>> >> 'nifi-site'   https://issues.apache.org/jira/browse/INFRA-10120
>> >>
>> >> Actions:
>> >> Once these two new git repositories are created i will move the
>> >> appropriate nifi-nar-maven-plugin items into it and terminate the
>> >> current directory.  Then I'll move the nifi-site directory content
>> >> into the new nifi-site repository and then delete the directory.
>> >>
>> >> Once that is sorted we can discuss whether we care to keep
>> >> develop/master or simply go to master as Ryan suggests.
>> >>
>> >> Thanks
>> >> Joe
>> >>
>> >> On Mon, Aug 10, 2015 at 5:13 PM, Joe Witt  wrote:
>> >>>
>> >>> Ryan
>> >>>
>> >>> Correct the latest code depends on latest nifi nar maven plugin.
>> >>>
>> >>> I would be absolutely fine personally with eliminating develop and just
>> >>> using master.  Given that the releases are tagged i personally dont get
>> >>> the
>> >>> value here vs the extra work required.
>> >>>
>> >>> Anybody feel strongly for keeping master and dev as they are and if so
>> >>> can
>> >>> you please state how the current model has helped you contribute or how
>> >>> the
>> >>> proposed model would not?
>> >>>
>> >&g

Re: How to delete docs for renamed processor

[Dan Bress]

Rick,
   Can you confirm whether the old processor name still shows up in the "new 
processor" UI when you try and drag a new processor on the graph?

   Also did you do a 'mvn clean install' on your nar, or just 'mvn install'?

Dan Bress
Software Engineer
ONYX Consulting Services


From: Rick Braddy 
Sent: Wednesday, September 9, 2015 3:00 PM
To: dev@nifi.apache.org
Subject: RE: How to delete docs for renamed processor

Thanks Mark.  Unfortunately, that didn't work - it's still there.

Rick

-Original Message-
From: Mark Payne [mailto:marka...@hotmail.com]
Sent: Wednesday, September 09, 2015 1:45 PM
To: dev@nifi.apache.org
Subject: RE: How to delete docs for renamed processor

Rick,

You can try deleting the nifi/work directory and restarting.

Thanks
-Mark


> From: rbra...@softnas.com
> To: dev@nifi.apache.org
> Subject: How to delete docs for renamed processor
> Date: Wed, 9 Sep 2015 18:42:05 +
>
> Hi,
>
> Hopefully a quick answer. I have renamed a custom processor and I still see 
> the processor by the old name showing up in the user docs. Is there a way to 
> delete the docs related to the old processor name?
>
> Thanks
> Rick
>

Re: How to delete docs for renamed processor

[Dan Bress]

Rick,
   It sounds like the nar contains both the old .class and the new .class

   Can you run "mvn clean install" on your nar, then look at the 
nar/META-INF/bundled-dependencies/your-processors.jar and see if it contains 
both the old class and the new class?

Dan Bress
Software Engineer
ONYX Consulting Services


From: Rick Braddy 
Sent: Wednesday, September 9, 2015 3:25 PM
To: dev@nifi.apache.org
Subject: RE: How to delete docs for renamed processor

Dan,

Yes. The old processor name still shows up in the UI when dragging a new 
processor onto the canvass, so it's not just the docs that are remaining.

"mvn -T C2.0 clean install" is the nar build command used.

Rick

-Original Message-
From: Dan Bress [mailto:dbr...@onyxconsults.com]
Sent: Wednesday, September 09, 2015 2:06 PM
To: dev@nifi.apache.org
Subject: Re: How to delete docs for renamed processor

Rick,
   Can you confirm whether the old processor name still shows up in the "new 
processor" UI when you try and drag a new processor on the graph?

   Also did you do a 'mvn clean install' on your nar, or just 'mvn install'?

Dan Bress
Software Engineer
ONYX Consulting Services


From: Rick Braddy 
Sent: Wednesday, September 9, 2015 3:00 PM
To: dev@nifi.apache.org
Subject: RE: How to delete docs for renamed processor

Thanks Mark.  Unfortunately, that didn't work - it's still there.

Rick

-Original Message-
From: Mark Payne [mailto:marka...@hotmail.com]
Sent: Wednesday, September 09, 2015 1:45 PM
To: dev@nifi.apache.org
Subject: RE: How to delete docs for renamed processor

Rick,

You can try deleting the nifi/work directory and restarting.

Thanks
-Mark


> From: rbra...@softnas.com
> To: dev@nifi.apache.org
> Subject: How to delete docs for renamed processor
> Date: Wed, 9 Sep 2015 18:42:05 +
>
> Hi,
>
> Hopefully a quick answer. I have renamed a custom processor and I still see 
> the processor by the old name showing up in the user docs. Is there a way to 
> delete the docs related to the old processor name?
>
> Thanks
> Rick
>

Re: How to delete docs for renamed processor

[Dan Bress]

Rick,
   Great. Can you drop that in the lib directory(and make sure you don't have 
any other versions of this jar in that lib directory), stop nifi, delete the 
work directory, and restart?  I think that will get you to where you want to be.

Dan Bress
Software Engineer
ONYX Consulting Services


From: Rick Braddy 
Sent: Wednesday, September 9, 2015 3:46 PM
To: dev@nifi.apache.org
Subject: RE: How to delete docs for renamed processor

Dan,

Ran the "mvn clean install" and inspected the resulting process jar file.  The 
old class is not present in the jar file using "jar tf processor.jar"

Rick

-Original Message-
From: Dan Bress [mailto:dbr...@onyxconsults.com]
Sent: Wednesday, September 09, 2015 2:37 PM
To: dev@nifi.apache.org
Subject: Re: How to delete docs for renamed processor

Rick,
   It sounds like the nar contains both the old .class and the new .class

   Can you run "mvn clean install" on your nar, then look at the 
nar/META-INF/bundled-dependencies/your-processors.jar and see if it contains 
both the old class and the new class?

Dan Bress
Software Engineer
ONYX Consulting Services


From: Rick Braddy 
Sent: Wednesday, September 9, 2015 3:25 PM
To: dev@nifi.apache.org
Subject: RE: How to delete docs for renamed processor

Dan,

Yes. The old processor name still shows up in the UI when dragging a new 
processor onto the canvass, so it's not just the docs that are remaining.

"mvn -T C2.0 clean install" is the nar build command used.

Rick

-Original Message-
From: Dan Bress [mailto:dbr...@onyxconsults.com]
Sent: Wednesday, September 09, 2015 2:06 PM
To: dev@nifi.apache.org
Subject: Re: How to delete docs for renamed processor

Rick,
   Can you confirm whether the old processor name still shows up in the "new 
processor" UI when you try and drag a new processor on the graph?

   Also did you do a 'mvn clean install' on your nar, or just 'mvn install'?

Dan Bress
Software Engineer
ONYX Consulting Services


From: Rick Braddy 
Sent: Wednesday, September 9, 2015 3:00 PM
To: dev@nifi.apache.org
Subject: RE: How to delete docs for renamed processor

Thanks Mark.  Unfortunately, that didn't work - it's still there.

Rick

-Original Message-
From: Mark Payne [mailto:marka...@hotmail.com]
Sent: Wednesday, September 09, 2015 1:45 PM
To: dev@nifi.apache.org
Subject: RE: How to delete docs for renamed processor

Rick,

You can try deleting the nifi/work directory and restarting.

Thanks
-Mark


> From: rbra...@softnas.com
> To: dev@nifi.apache.org
> Subject: How to delete docs for renamed processor
> Date: Wed, 9 Sep 2015 18:42:05 +
>
> Hi,
>
> Hopefully a quick answer. I have renamed a custom processor and I still see 
> the processor by the old name showing up in the user docs. Is there a way to 
> delete the docs related to the old processor name?
>
> Thanks
> Rick
>

Re: 1.0.0 Branch Guidance

[Dan Bress]

Aldrin,
   I definitely like the idea of creating separate branches for the 0.3.X work 
and the 1.X.X work.  

   My thoughts would be to make 'master' the 1.X version, and make a branch for 
the 0.3.X work.  The reason being, I would imagine most of the work the 
community is doing should be slated for 1.X, where as less work(e.g. bug fixes) 
be done in the 0.3.X branch.  And by making master 1.0.0 it nudges people in 
that direction.  Also, I'm assuming that 0.3.X will just be bug fixes at this 
point, and our next 'major' release will be 1.0.0.   Is that a fair assumption 
to make?  If not, I might be more inclined to agree with your original 
suggestion, although keeping that long living branch up to date with all the 
changes from master might be a maintenance nightmare.

Dan Bress
Software Engineer
ONYX Consulting Services


From: Aldrin Piri 
Sent: Thursday, September 24, 2015 10:49 AM
To: dev@nifi.apache.org
Subject: 1.0.0 Branch Guidance

All,

We are starting to receive more items that are "breaking" changes
(deprecation of components and code being those that immediately come to
mind) and are accordingly scheduled for a 1.0.0 release.  I would like to
solicit the community for best practices on the introduction and
maintenance of a 1.0.0 branch so we can do so in a practical manner.

There are a few PRs/patches that are sitting in limbo because we do not
have an appropriate place to put them and would very much like to be able
to incorporate those changes and close them in lieu of waiting until master
reaches that juncture.

Any input on caveats, items to note, and any other items to be mindful of
is greatly appreciated.

Thanks!

Re: 1.0.0 Branch Guidance

[Dan Bress]

OK, sounds good to me.  If I had to choose between keeping most of the 
communities work 'parked' as PullRequests and patches vs. merged into a 1.X 
branch that we keep up to date with master, I'd vote for 1.X branch.

Dan Bress
Software Engineer
ONYX Consulting Services


From: Aldrin Piri 
Sent: Tuesday, September 29, 2015 8:48 AM
To: dev@nifi.apache.org
Subject: Re: 1.0.0 Branch Guidance

Dan,

I don't think we are at the point where 1.0 is our next release.  The items
to be included for that release as per the feature proposals (whether
directly as a feature or as an implicit dependency for one or more of those
features) are some considerable efforts and while there may not be many
releases between 0.3.0 and that point, it will likely be too long to go
without any at all.

Certainly agree on the long living branch being an effort of itself, but
may be the course we have to take to keep the project moving.

On Tue, Sep 29, 2015 at 8:41 AM, Dan Bress  wrote:

> Aldrin,
>I definitely like the idea of creating separate branches for the 0.3.X
> work and the 1.X.X work.
>
>My thoughts would be to make 'master' the 1.X version, and make a
> branch for the 0.3.X work.  The reason being, I would imagine most of the
> work the community is doing should be slated for 1.X, where as less
> work(e.g. bug fixes) be done in the 0.3.X branch.  And by making master
> 1.0.0 it nudges people in that direction.  Also, I'm assuming that 0.3.X
> will just be bug fixes at this point, and our next 'major' release will be
> 1.0.0.   Is that a fair assumption to make?  If not, I might be more
> inclined to agree with your original suggestion, although keeping that long
> living branch up to date with all the changes from master might be a
> maintenance nightmare.
>
> Dan Bress
> Software Engineer
> ONYX Consulting Services
>
> 
> From: Aldrin Piri 
> Sent: Thursday, September 24, 2015 10:49 AM
> To: dev@nifi.apache.org
> Subject: 1.0.0 Branch Guidance
>
> All,
>
> We are starting to receive more items that are "breaking" changes
> (deprecation of components and code being those that immediately come to
> mind) and are accordingly scheduled for a 1.0.0 release.  I would like to
> solicit the community for best practices on the introduction and
> maintenance of a 1.0.0 branch so we can do so in a practical manner.
>
> There are a few PRs/patches that are sitting in limbo because we do not
> have an appropriate place to put them and would very much like to be able
> to incorporate those changes and close them in lieu of waiting until master
> reaches that juncture.
>
> Any input on caveats, items to note, and any other items to be mindful of
> is greatly appreciated.
>
> Thanks!
>

Re: Source code for Version 0.3.0

[Dan Bress]

I think a tag for each release signed by the person who originally released it 
would make the most sense to anyone looking at our codebase.

Dan Bress
Software Engineer
ONYX Consulting Services


From: Sean Busbey 
Sent: Friday, October 2, 2015 11:35 AM
To: dev@nifi.apache.org
Subject: Re: Source code for Version 0.3.0

If we're going with tags, I'd love one for each previous release.

On Fri, Oct 2, 2015 at 7:48 AM, Adam Taft  wrote:
> Just bumping this conversation.  Did we end up addressing this?  Are we
> going for a signed release tag?  If so, does it make sense for the 0.3.0
> tag to be signed by the releasor (I believe Matt Gilman)?  Or maybe just an
> unsigned tag?
>
> Thanks,
>
> Adam
>
>
> On Mon, Sep 21, 2015 at 2:28 PM, Joe Witt  wrote:
>
>> Looks fairly straightforward to sign a release [1].
>>
>> What is the workflow you'd suggest?  Can we keep our current process
>> and once the vote is done just add a step to make a new identical (but
>> signed) tag with a name that doesn't include '-RC#'?
>>
>> I'm good with that.  I understand why the RC# throws folks off so
>> happy to sort this out.
>>
>> [1] http://gitready.com/advanced/2014/11/02/gpg-sign-releases.html
>>
>> On Mon, Sep 21, 2015 at 12:42 PM, Ryan Blue  wrote:
>> > +1 for a nifi-0.3.0 release tag. Signed is even better, but I don't think
>> > I'd mind if it weren't signed.
>> >
>> > rb
>> >
>> >
>> > On 09/21/2015 06:35 AM, Sean Busbey wrote:
>> >>
>> >> The pattern I've liked the most on other projects is to create a
>> >> proper release tag, signed by the RM on passage of the release vote. I
>> >> don't recall off-hand what the phrasing was in the VOTE thread (if
>> >> any).
>> >>
>> >> On Mon, Sep 21, 2015 at 8:13 AM, Adam Taft  wrote:
>> >>>
>> >>> What's the thoughts on creating a proper 0.3.0 tag, as would be
>> >>> traditional
>> >>> for a final release?  It is arguably a little confusing to only have
>> the
>> >>> RC
>> >>> tags, when looking for the final release.  I found this head scratching
>> >>> for
>> >>> 0.2.0 as well.
>> >>>
>> >>> Adam
>> >
>> >
>> >
>> >
>> > --
>> > Ryan Blue
>> > Software Engineer
>> > Cloudera, Inc.
>>



--
Sean

Fulfilling a pull request

[dan bress]

Apache NiFi Committers,
   I am working on fulfilling a pull request(NIFI-774), and wanted to make
sure I am doing it right.

   These are the steps that I plan on following

   1) git remote add yu https://github.com/yu-iskw/nifi.git
   2) git pull yu
   3) git checkout master
   4) git merge --no-ff yu/NIFI-774
   5) clean install install -Pcontrib-check
   6) double check code and inspect release
   7) git push origin

How does that sound to you guys?

Thanks,
Dan

Re: Fulfilling a pull request

[dan bress]

Whoops, step 5 should be
 5) mvn clean install -Pcontrib-check

On Mon, Oct 12, 2015 at 10:00 AM dan bress  wrote:

> Apache NiFi Committers,
>I am working on fulfilling a pull request(NIFI-774), and wanted to make
> sure I am doing it right.
>
>These are the steps that I plan on following
>
>1) git remote add yu https://github.com/yu-iskw/nifi.git
>2) git pull yu
>3) git checkout master
>4) git merge --no-ff yu/NIFI-774
>5) clean install install -Pcontrib-check
>6) double check code and inspect release
>7) git push origin
>
> How does that sound to you guys?
>
> Thanks,
> Dan
>

"Integration" style Unit Tests

[dan bress]

Devs,
   While working on integrating and testing the work Yu did for
NIFI-774/DeleteS3Object, I noticed that a few of the unit tests for the
processors in that AWS bundle(Put and Fetch S3) actually interact with S3
directly, and were marked as @Ignore.  If I wanted to un @Ignore them and
actually run them I needed to set up an AWS account, then copy the
credentials into an aws-credentials.properties file and put that in my home
directory to get the tests to pass.  This dashed my hopes of a relatively
simple merge and turned it into a bit of work for me.  I'm not blaming Yu
or anyone for this, just wanted to open up a discussion on better ways of
solving this.

Problem:
@Ignore'd tests don't get run, probably ever.  Why?  Because running them
is a pain in the butt, I agree with NIFI-438[1] "If tests could talk they
would say don't @Ignore me".  I appreciate that there are special
circumstances for using this, but it would probably benefit everyone if we
sure we used it only for the truly special circumstances.


Solutions:
a. Run these tests using the failsafe plugin[2] instead of the surefire
plugin.  This way they get run every time, and if they fail that
information gets reported but it does not stop the build.
b. Mock out the service(I appreciate that this may not always be possible)
c. Provide instructions somewhere so that someone with no experience with
these processors/tests can run them.

Anyone have thoughts on this?
My votes would be B, and if that is not possible A and C.

Thanks,
Dan

[1] https://issues.apache.org/jira/browse/NIFI-438
[2] https://maven.apache.org/surefire/maven-failsafe-plugin/

Re: Fulfilling a pull request

[dan bress]

Aldrin,
   Awesome, I appreciate your perspective on this.  I'll check out option
#2.  Thanks!

Dan

On Mon, Oct 12, 2015 at 12:42 PM Aldrin Piri  wrote:

> That way seems reasonable and should work.  The only negative, and I use
> the term loosely, is the continuous addition of remotes as new contributors
> provide submissions (certainly a nice problem to have!).  There are a few
> other ways you can tackle this to combat adding a remote for each user that
> may come along, and I'll list two I've encountered as a quick reference:
>
> * Grabbing a patch file:  You are able to append .patch onto any pull
> request URL to get a patch file that you can apply in the fashion to which
> you are accustomed.  [1]
> * Getting PRs directly from Github via [2] or [3]
>
> I primarily use the method of [2] myself.  There are some caveats about how
> this functions in general listed in the comments, but for the case of
> working with NiFi and treating github as another remote, has been a pretty
> streamlined process for me.  [3] is mechanically quite similar to [2], but
> does a full path to the PR in the remote.
>
> [1] https://github.com/blog/967-github-secrets#diff-patch
> [2] https://gist.github.com/piscisaureus/3342247
> [3]
>
> https://help.github.com/articles/checking-out-pull-requests-locally/#modifying-an-inactive-pull-request-locally
>
> On Mon, Oct 12, 2015 at 10:36 AM, dan bress  wrote:
>
> > Whoops, step 5 should be
> >  5) mvn clean install -Pcontrib-check
> >
> > On Mon, Oct 12, 2015 at 10:00 AM dan bress  wrote:
> >
> > > Apache NiFi Committers,
> > >I am working on fulfilling a pull request(NIFI-774), and wanted to
> > make
> > > sure I am doing it right.
> > >
> > >These are the steps that I plan on following
> > >
> > >1) git remote add yu https://github.com/yu-iskw/nifi.git
> > >2) git pull yu
> > >3) git checkout master
> > >4) git merge --no-ff yu/NIFI-774
> > >5) clean install install -Pcontrib-check
> > >6) double check code and inspect release
> > >7) git push origin
> > >
> > > How does that sound to you guys?
> > >
> > > Thanks,
> > > Dan
> > >
> >
>

Re: "Integration" style Unit Tests

[dan bress]

I agree with what JoeS is saying.   Having these core integration aspects
of the system tested in an automated way only when needed(like before a
release) would be a huge plus for NiFi.

I think Sean provides a great mechanism for implementing this with
Categories.  Sean, can you speak to the build server bit?  Is this
something we have?  Is this something we are working towards?

We would still need a mechanism for capturing the credentials required to
run tests that integrate with remote services.  Maybe this lives on the
build server?  Maybe we have a stub .properties file that shows credentials
you need to interact with AWS, kafka, Twitter, etc..

Also if we can tag TestJdbcHugeStream with @SlowTests so I don't have to
wait for it every time I run tests then I will be t h r i l l e d.

Dan

On Wed, Oct 14, 2015 at 3:19 PM Sean Busbey  wrote:

> JUnit Categories would work perfectly for this:
>
> https://github.com/junit-team/junit/wiki/Categories
>
> You can group either at the class test layer or on individual tests.
>
> we can then run the categories that require external resources on
> build servers that have the needed instances present.
>
> On Wed, Oct 14, 2015 at 1:46 PM, Joe Skora  wrote:
> > I agree with you that these shouldn't run as normal unit tests.  But, I'm
> > worried that it means this functionality will not be regularly tested as
> > external APIs change.  Mocking an external API doesn't validate much, if
> I
> > don't understand the external resource I'll probably make the same
> mistakes
> > in the mock as I do in the client, so testing against the real external
> > resource is the only way to know things work.
> >
> > Perhaps a set of test scripts are needed to automate groups of @ignored
> > tests that share an external dependency such as an AWS script, a Hadoop
> > script, a Kafka script, etc?
> >
> > I have never used JUnit's filtering capability, but it might be possible
> to
> > use that in combination with a Maven profile to automate those test
> groups
> > instead of requiring scripts be maintained.
> >
> > Just brainstorming.
> >
> > Regards,
> > JoeS
> >
> > On Mon, Oct 12, 2015 at 8:22 PM, Joe Witt  wrote:
> >
> >> Dan,
> >>
> >> Yeah this is a good housekeeping item to bring up.  My view here is
> >> that B is the only answer.  Unit tests should not be calling out to
> >> external services - period.
> >>
> >> Thanks
> >> Joe
> >>
> >> On Mon, Oct 12, 2015 at 8:17 PM, dan bress  wrote:
> >> > Devs,
> >> >While working on integrating and testing the work Yu did for
> >> > NIFI-774/DeleteS3Object, I noticed that a few of the unit tests for
> the
> >> > processors in that AWS bundle(Put and Fetch S3) actually interact
> with S3
> >> > directly, and were marked as @Ignore.  If I wanted to un @Ignore them
> and
> >> > actually run them I needed to set up an AWS account, then copy the
> >> > credentials into an aws-credentials.properties file and put that in my
> >> home
> >> > directory to get the tests to pass.  This dashed my hopes of a
> relatively
> >> > simple merge and turned it into a bit of work for me.  I'm not
> blaming Yu
> >> > or anyone for this, just wanted to open up a discussion on better
> ways of
> >> > solving this.
> >> >
> >> > Problem:
> >> > @Ignore'd tests don't get run, probably ever.  Why?  Because running
> them
> >> > is a pain in the butt, I agree with NIFI-438[1] "If tests could talk
> they
> >> > would say don't @Ignore me".  I appreciate that there are special
> >> > circumstances for using this, but it would probably benefit everyone
> if
> >> we
> >> > sure we used it only for the truly special circumstances.
> >> >
> >> >
> >> > Solutions:
> >> > a. Run these tests using the failsafe plugin[2] instead of the
> surefire
> >> > plugin.  This way they get run every time, and if they fail that
> >> > information gets reported but it does not stop the build.
> >> > b. Mock out the service(I appreciate that this may not always be
> >> possible)
> >> > c. Provide instructions somewhere so that someone with no experience
> with
> >> > these processors/tests can run them.
> >> >
> >> > Anyone have thoughts on this?
> >> > My votes would be B, and if that is not possible A and C.
> >> >
> >> > Thanks,
> >> > Dan
> >> >
> >> > [1] https://issues.apache.org/jira/browse/NIFI-438
> >> > [2] https://maven.apache.org/surefire/maven-failsafe-plugin/
> >>
>
>
>
> --
> Sean
>

Re: [ANNOUNCE] New Apache NiFi Committer Ricky Saltzer

[dan bress]

Welcome Ricky!  Thanks for your contributions and I look forward to you
pushing Apache NiFi forward!

On Wed, Oct 21, 2015 at 3:04 PM Tony Kurc  wrote:

> NiFi Community!
>
> Great news!
>
> On behalf of the Apache NiFI PMC, I am very pleased to announce that Ricky
> Saltzer has accepted the PMC's invitation to become a committer on the
> Apache NiFi project. We greatly appreciate all of Ricky's hard work and
> generous contributions to the project. We look forward to his continued
> involvement in the project.
>
> Welcome Ricky, and congratulations!
>
> Tony
>

Re: [DISCUSS] roadmap for the next 6-12 months

[dan bress]

Looks well thought out to me!  Thanks to all who put the effort into this!

On Fri, Jan 8, 2016 at 6:29 AM Joe Witt  wrote:

> Hello NiFi Community
>
> Please review the proposed roadmap and suggest things to add, remove,
> update, prioritize differently.
>
> Over the past year we have had great contributions that improved the
> quality of NiFi and the community.  As we know there are many JIRAs
> related to improvements or new features.  However, it can be difficult
> for new and existing community members to know when to jump in, how to
> help, or when things they are interested in might be able to happen.
> So among other things we should provide and maintain a documented
> roadmap to help this.
>
> The following suggested timeline follows pretty strictly against our
> previously stated objective of having releases roughly every 6 weeks.
> We do not need to stay strict to this but a time bound (as opposed to
> feature bound) release cycle does help provide a regular cadence to
> follow and helps give contributors a sense of when their efforts might
> end up in a release.
>
> The items below are largely just focused on some of the major muscle
> movements needed at a framework level.  There will of course be many
> many more JIRAs with each releas.
>
> NiFi 0.5.0 (Feb 5)
>
>   State Management
> https://cwiki.apache.org/confluence/display/NIFI/State+Management
>   Kerberos
>
> https://cwiki.apache.org/confluence/display/NIFI/Pluggable+Authentication
>   Interactive Queue
>
> https://cwiki.apache.org/confluence/display/NIFI/Interactive+Queue+Management
>   Scripting Support
> https://issues.apache.org/jira/browse/NIFI-210
>
> NiFi 0.6.0 (Mar 18)
>
>   Deterministic Template Export
> https://issues.apache.org/jira/browse/NIFI-826
>   Schema/Format Editor (JSON to JSON,…?)
> Numerous JIRAs related to format/schema conversion.  An example
> implementation:
>
> https://github.com/fsauer65/NiFi-Extensions/tree/master/nifi-jsontransform-bundle
>
> NiFi 0.7.0 (Apr 29)
>
> NiFi 1.0.0 (Jun 17)
>
>   HA Cluster Management
> https://cwiki.apache.org/confluence/display/NIFI/Clustering+Redesign
>   HA Data
>
> https://cwiki.apache.org/confluence/display/NIFI/High+Availability+Processing
>   Multi-Tenant Authorization
>
> https://cwiki.apache.org/confluence/display/NIFI/Multi-Tentant+Dataflow
>   Improved Authorization API
>
> https://cwiki.apache.org/confluence/display/NIFI/Pluggable+Authentication
> NiFi's current authority provider gathers data then makes a decision.
> We should instead support PDP/PEP model [4].
>   Modernized UX
>
> https://cwiki.apache.org/confluence/display/NIFI/Redesign+User+Interface
>   Dependent Properties
> https://issues.apache.org/jira/browse/NIFI-1121
>
> NiFi 1.1.0 (Sep?)
>
>   Template/Extension Registry
>
> https://cwiki.apache.org/confluence/display/NIFI/Extension%2C+Template%2C+Dataset+Registry
>   Variable Registry
> https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry
>
> The timing of these things is pretty important as far as
> interdependencies and how long they'll take to develop and validate.
> State management in 050 sets up an important way to have cluster-wide
> shared state and introduces zookeeper formally into NiFi which
> obviously will be important for the distributed systems behaviors
> we'll need as we restructure how clustering works and introduce data
> replication for HA data processing.  The 060 and 070 releases are
> intentionally quite light to allow for very focused work on a parallel
> branch of 1.x which probably should start right on the heels of 050.
> The 1.0 release will introduce some very key features, offer a chance
> to remove deprecated classes, ensure all APIs are extremely clearly
> documented for stability and audience, allow us to introduce UI/UX
> changes, and so on.
>
> Now, of course this is simply a proposal and these timings may not
> work out.  But this should serve as a good basis for discussion and
> consensus forming.
>
> As we progress I plan to update the roadmap [1], product requirements
> [2], and feature proposals [3] to reflect these things a bit more
> clearly than they are now.
>
> We should also have a discussion on how long we should be committed to
> supporting the 0.x line and what that means.  We need to document a
> commitment for the community.
>
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=58851850
> [2] https://cwiki.apache.org/confluence/display/NIFI/Product+requirements
> [3]
> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals
> [4] https://en.wikipedia.org/wiki/Common_Open_Policy_Service
>
> Thanks
> Joe
>

Re: Component documentation improvements

[dan bress]

I just tossed some comments on the wikipage.

TL;DR I took a look at generating docs as part of the build months ago.  I
think its doable, and someone should pursue it.

On Mon, Jan 18, 2016 at 2:38 PM Joe Witt  wrote:

> Oleg,
>
> "Yes, there will be breaking changes. Its not a question of IF, but
> rather WHEN."
>
> I disagree.  It is always a question of IF.
>
> We have to be extremely judicious in the use of breaking changes and
> we owe the user/developer based excellent justification in such cases.
>
> I will comment on the Wiki page for the substance of this particular
> proposal (documentation generation).
>
> Joe
>
> On Mon, Jan 18, 2016 at 1:22 PM, Oleg Zhurakousky
>  wrote:
> > Josh
> >
> > FWIW, let’s use WIKI comments to maintain a discussion. It will be
> simpler in the end to compile a resolution and move on.
> >
> > Yet, I’ll reply here anyway.
> > Yes, there will be breaking changes. Its not a question of IF, but
> rather WHEN.
> > What we can do is make it less painful by introducing certain changes
> gradually with deprecations and clear communication to the community on
> what is about to change. Other mechanics could be applied here as well, but
> before we get into the mechanics, I’d like to see if there are any more
> ideas, concerns etc, so we can have a join resolution as to what is a
> sustainable documentation model of the future NiFi, then we can figure out
> how to get there.
> >
> > Cheers
> > Oleg
> >
> >> On Jan 18, 2016, at 1:08 PM, Joshua Davis 
> wrote:
> >>
> >> Oleg,
> >>
> >> Interesting document, what impact would it have on existing
> installations
> >> of NIFI?
> >>
> >> What would be the upgrade path for Custom Processors?
> >>
> >> Are we breaking compatibility with the previous way of doing
> documentation?
> >>
> >> Why not create a simple content repository that can hold the
> documentation
> >> information?
> >>
> >> Is there a plan for multiple languages?
> >>
> >> Joshua Davis
> >> Senior Consultant
> >> Hortonworks Professional Services
> >> (407)476-6752
> >>
> >>
> >>
> >>
> >>
> >>
> >> On 1/18/16, 12:09 PM, "Oleg Zhurakousky" 
> >> wrote:
> >>
> >>> Guys
> >>>
> >>> I¹ve just finished initial draft of the proposal to improve our
> component
> >>> documentation mechanisms (e.g., Processors, ControllerServices etc).
> >>>
> https://cwiki.apache.org/confluence/display/NIFI/Component+documentation+i
> >>> mprovements
> >>> Please give it a read and let¹s get discussion going.
> >>>
> >>> Cheers
> >>> Oleg
> >>
> >>
> >
>

Re: Javadoc

[dan bress]

I back putting NiFi Javadocs on the website, as it will provide a good
place for new developers to learn how to extend NiFi.

I believe we have a couple of Jira tickets asking for this [1] [2]

Joe, mvn site can definitely do it, whether that is the 'best' way or not
is debatable, since mvn site is usually very very very slow.  Another
option is to call the javadoc plugin directly, i think either the javadoc
goal or the aggregate goal is what we want.

[1] https://issues.apache.org/jira/browse/NIFI-943
[2] https://issues.apache.org/jira/browse/NIFI-445


On Tue, Jan 19, 2016 at 5:49 AM Joe Witt  wrote:

> What is the best way to do this?  mvn site?
>
> On Tue, Jan 19, 2016 at 8:41 AM, Sean Busbey  wrote:
> > Hi Devin!
> >
> > Sorry for the gap. We're tracking the need for a web accessible copy
> > of our API javadocs in NIFI-943. I don't believe anyone has taken up
> > the task yet, so if you're looking to get more involved in the project
> > we'd love the help!
> >
> > On Sun, Jan 17, 2016 at 1:38 PM, Devin Fisher
> >  wrote:
> >> Might be a silly question but I can't find a link the Javadocs anywhere
> on
> >> your website. I've googled for it and only find the reference to the
> >> Javadocs in the NiFi Developer’s Guide
> >> . I'm hopeful that I'm
> just
> >> not seeing it. If not, what is the recommended way to get a hold of it?
> >>
> >> Also, thanks in advanced.
> >>
> >> Devin
>

Re: Auto-Organize Layout

[dan bress]

Maybe not exactly "auto-layout" but I would back a notion of having the
components snap to a coarser grain grid than what we currently have.
Sometimes I care a lot about having everything line up in the graph
horizontally and vertically, and it always takes a long time to achieve
this.

I could see this being achieved by snapping the component to the same spot
horizontally as the component above it when you move it underneath another
component.  Or some magical "auto snap" button that does its best to align
everything with its nearest neighbors.

On Tue, Jan 19, 2016 at 12:37 PM Ryan H  wrote:

> I like your idea Rob, that would help with lining up relationships too
> (straight lines).
>
> On Matt's note, I don't think there should be a "standard" either, although
> best practices are always out there.
>
> On Matt's note of putting failures up above processes, we do that too.
> Totally depends on who made the flow first.  Sometimes, people don't even
> follow a convention in the same flow.xml file.
>
> For these reasons, I'd recommend alternate views to the flow.
>
> We have a couple projects that just allow you to rearrange a node-based
> graph, based on your preference, hierarchy, circular, pyramid, etc.
>
> Applying this to NiFi, having a couple different default auto-layout
> options that you can swap your current view to, but NOT change the original
> flow, would be nice.
>
> It would let you walk into someone else's, potentially large, dataflow and
> have a familiar way to view the flow.
>
> Ryan
>
>
> On Tue, Jan 19, 2016 at 2:03 PM, Rob Moran  wrote:
>
> > I agree with Matt's points. I was just replying with something similar
> > basically saying I think trying to set a standard would not be
> > well-received.
> >
> > I believe what could be more useful are layout tools that would help
> users
> > place components to help achieve their preferred layouts. For example,
> the
> > ability to align (left, right, center) components
> > or horizontally/vertically distribute components evenly. Other features
> > such as snap-to and/or smart-guides could make it easier for users to
> > follow their organization's best practices when designing a flow.
> >
> > Rob
> >
> > On Tue, Jan 19, 2016 at 1:49 PM, Matthew Clarke <
> matt.clarke@gmail.com
> > >
> > wrote:
> >
> > > Ryan,
> > >
> > >   Setting a standard is a difficult thing to do.  The
> complexity
> > > that can exist in many flows would make enforcing a standard difficult.
> > The
> > > first example you provide of success to points right while failures
> point
> > > up is not recommended. It would be better to have failures point down
> > since
> > > it is common to put labels over processor(s). Any relationships
> pointing
> > up
> > > would pass through these labels making both the relationship box and
> the
> > > label hard to read.  It is often coomon to see flows designed with a
> > > combination of left to right and top to bottom design.
> > >
> > > Matt
> > >
> > > On Tue, Jan 19, 2016 at 12:07 PM, Ryan H 
> > > wrote:
> > >
> > > > Hi Rob,
> > > > Yea we did, it was at the end of the meeting.
> > > >
> > > > I think it would be useful to have a couple default type views
> that
> > > > help standardize flow layout across the community.
> > > >
> > > > For example, when we organize processors left-to-right, failure
> > > > relationships always point up, and success always point right.
> > > > Alternatively, when we organize processors up-and-down, failure
> > > > relationships always point left, and successes always point down.
> > > >
> > > > Of course, in some of these scenarios there are processors that
> > have
> > > > more than 1 success relationship, but that's when a good layout
> library
> > > > would come into play.
> > > >
> > > > What do you think?
> > > >
> > > > On Tue, Jan 19, 2016 at 11:10 AM, Rob Moran 
> wrote:
> > > >
> > > > > Ryan - I think we spoke briefly (at a very high level) about this
> at
> > a
> > > > > prior meetup. What alternate views did you have in mind, and in
> what
> > > ways
> > > > > do you think they'd be useful?
> > > > >
> > > > > Rob
> > > > >
> > > > > On Tue, Jan 19, 2016 at 10:56 AM, Ryan H <
> > rhendrickson.w...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > It'd be pretty awesome if NiFi provided the ability to
> > auto-organize
> > > a
> > > > > > layout.
> > > > > >
> > > > > > Maybe even just a auto-organized layout that doesn't change the
> > > > flow.xml,
> > > > > > just an alternate view.
> > > > > >
> > > > > > Looking at these demos here: http://js.cytoscape.org/#demos
> > > > > >
> > > > > > Ryan
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Are we thinking about Penalization all wrong?

[dan bress]

Mark,
   I agree with all the points you mention about penalization being
confusing, and I think the ability to apply a penalty to Flowfile's outside
of a processor is a clearer way to express what is happening.

   I worry that having the penalty be a property of the connection would
also be confusing.  To me, penalizing a FlowFile is an action you do to a
FlowFile.  In my head, connections don't do actions on FlowFile, they just
sort them and move them along.  I might find it confusing that the
connection is "doing things" to the flow files, unless there was some kind
of visual cue as to what was going on.  Kind of like how people have
brought up that the "expire" concept is a little confusing, because of the
lack of visual cue.

So when I started typing this email I was thinking we should have a new
concept of a "penalizer" that's kind of like a processor but just puts a
penalty on a flow file.  After typing it, that might be a new construct
that isn't really needed, and I'm OK with this being put on a connection,
but I would like there to be a visual cue on the connection indicating that
it is penalizing flow files.

On Thu, Jan 28, 2016 at 8:34 AM Mark Payne  wrote:

> All,
>
> I've been thinking about how we handle the concept of penalizing
> FlowFiles. We've had a lot of questions
> lately about how penalization works & the concept in general. Seems the
> following problems exist:
>
> - Confusion about difference between penalization & yielding
> - DFM sees option to configure penalization period on all processors, even
> if they don't penalize FlowFiles.
> - DFM cannot set penalty duration in 1 case and set a different value for
> a different case (different relationship, for example).
> - Developers often forget to call penalize()
> - Developer has to determine whether or not to penalize when building a
> processor. It is based on what the developer will
> think may make sense, but in reality DFM's sometimes want to penalize
> things when the processor doesn't behave that way.
>
> I'm wondering if it doesn't make sense to remove the concept of
> penalization all together from Processors and instead
> move the Penalty Duration so that it's a setting on the Connection. I
> think this would clear up the confusion and give the DFM
> more control over when/how long to penalize. Could set to the default to
> 30 seconds for self-looping connections and no penalization
> for other connections.
>
> Any thoughts?
>
> Thanks
> -Mark

Re: Re: Re: Processor additional documentation

[dan bress]

Uwe,
   No, the index.html is generated for you.  additionalDetails.html is your
responsibility only if you feel like the generated index.html doesn't fully
describe your processor.

   I would guess 80% of the included processors do not have
additionalDetails.html.  If you haven't browsed here [1] at examples of the
generated index.html and user supplied additionalDetails.html, it might
clear things up.

[1] https://nifi.apache.org/docs.html

Dan

On Fri, Mar 18, 2016 at 11:08 AM Uwe Geercken  wrote:

> Dan,
>
> but maybe I have a wrong understanding: do I have to create an index.html
> file? Currently I have only created an additionalDetails.html file.
>
> I will also try to reduce the html code to a minimum and see if it is a
> problem with my code.
>
> Bye,
>
> Uwe
>
> > Gesendet: Freitag, 18. März 2016 um 19:03 Uhr
> > Von: "dan bress" 
> > An: dev@nifi.apache.org
> > Betreff: Re: Re: Processor additional documentation
> >
> > Uwe,
> >No its not a problem to have both index.html and
> additionalDetails.html
> >  The NiFi framework generates nearly all of the documentation for your
> > processor for you.  It will generate information about the properties and
> > relationships your processor exposes to its users.  If you need to
> express
> > more about your processor, then that is where additionalDetails.html
> comes
> > into play.  For example, if your processor uses a custom query language.
> >
> > Generated index.html example:
> >
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.attributes.UpdateAttribute/index.html
> >
> > additionalDetails.html example:
> >
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.attributes.UpdateAttribute/additionalDetails.html
> >
> > On Fri, Mar 18, 2016 at 10:54 AM Uwe Geercken 
> wrote:
> >
> > > Bryan,
> > >
> > > all looks ok. I looked into the nifi-home/work/docs folder. There is
> > > nothing but a components folder. Inside there is a folder for my
> processor:
> > > com.datamelt.nifi.test.TemplateProcessor and inside the folder there
> is a
> > > file index.html and it contains the code of my additionalDetails.html
> file.
> > >
> > > when I open the file in the web browser it looks good. I looked at
> other
> > > index.html files and they look similar.
> > >
> > > but I noted that some folders have an inde.html file AND an
> > > additionalDetails.html file. maybe that is the problem?
> > >
> > > greetings,
> > >
> > > Uwe
> > >
> > >
> > >
> > > Gesendet: Freitag, 18. März 2016 um 16:18 Uhr
> > > Von: "Bryan Bende" 
> > > An: dev@nifi.apache.org
> > > Betreff: Re: Processor additional documentation
> > > Hi Uwe,
> > >
> > > Do you have the additionalDetails.html file in your processors jar
> project,
> > > under src/main/resources?
> > >
> > > Similar to this:
> > >
> > >
> https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/src/main/resources
> > >
> > > The expected project structure is described here:
> > >
> > >
> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-ExampleProcessorBundleStructure[https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-ExampleProcessorBundleStructure]
> <https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-ExampleProcessorBundleStructure[https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions%23MavenProjectsforExtensions-ExampleProcessorBundleStructure]>
> > > <
> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-ExampleProcessorBundleStructure[https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions%23MavenProjectsforExtensions-ExampleProcessorBundleStructure]
> >
> > >
> > > If you think that part is setup correctly, can you check under
> > > nifi_home/work/docs and see if
> com.datamelt.nifi.test.TemplateProcessor is
> > > there?
> > >
> > > -Bryan
> > >
> > > On Fri, Mar 18, 2016 at 11:04 AM, Uwe Geercken 
> > > wrote:
> > >
> > > >
> > > > Hello,
> > > >
> > > > I am writing my first processor. As described in the documentation, I
> > > have
> > > > added an HTML file to be used when the user selects "Usage":
> > > >
> > > > docs/com.datamelt.nifi.test.TemplateProcessor/additionalDetails.html
> > > >
> > > > This is located in the root or the Processors nar file.
> > > >
> > > > The processor class is this:
> > > >
> > > > com/datamelt/nifi/test/TemplateProcessor.class
> > > >
> > > > The processor works, but selecting "Usage" won't show my HTML file.
> > > >
> > > > I understood that I write the HTML file and Nifi will picks it up
> when it
> > > > starts. Or is this not true?
> > > >
> > > > Thanks for feedback,
> > > >
> > > > Uwe
> > > >
> > >
> >
>

Re: Re: Processor additional documentation

[dan bress]

Uwe,
   No its not a problem to have both index.html and additionalDetails.html
 The NiFi framework generates nearly all of the documentation for your
processor for you.  It will generate information about the properties and
relationships your processor exposes to its users.  If you need to express
more about your processor, then that is where additionalDetails.html comes
into play.  For example, if your processor uses a custom query language.

Generated index.html example:
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.attributes.UpdateAttribute/index.html

additionalDetails.html example:
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.attributes.UpdateAttribute/additionalDetails.html

On Fri, Mar 18, 2016 at 10:54 AM Uwe Geercken  wrote:

> Bryan,
>
> all looks ok. I looked into the nifi-home/work/docs folder. There is
> nothing but a components folder. Inside there is a folder for my processor:
> com.datamelt.nifi.test.TemplateProcessor and inside the folder there is a
> file index.html and it contains the code of my additionalDetails.html file.
>
> when I open the file in the web browser it looks good. I looked at other
> index.html files and they look similar.
>
> but I noted that some folders have an inde.html file AND an
> additionalDetails.html file. maybe that is the problem?
>
> greetings,
>
> Uwe
>
>
>
> Gesendet: Freitag, 18. März 2016 um 16:18 Uhr
> Von: "Bryan Bende" 
> An: dev@nifi.apache.org
> Betreff: Re: Processor additional documentation
> Hi Uwe,
>
> Do you have the additionalDetails.html file in your processors jar project,
> under src/main/resources?
>
> Similar to this:
>
> https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/src/main/resources
>
> The expected project structure is described here:
>
> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-ExampleProcessorBundleStructure[https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-ExampleProcessorBundleStructure]
> 
>
> If you think that part is setup correctly, can you check under
> nifi_home/work/docs and see if com.datamelt.nifi.test.TemplateProcessor is
> there?
>
> -Bryan
>
> On Fri, Mar 18, 2016 at 11:04 AM, Uwe Geercken 
> wrote:
>
> >
> > Hello,
> >
> > I am writing my first processor. As described in the documentation, I
> have
> > added an HTML file to be used when the user selects "Usage":
> >
> > docs/com.datamelt.nifi.test.TemplateProcessor/additionalDetails.html
> >
> > This is located in the root or the Processors nar file.
> >
> > The processor class is this:
> >
> > com/datamelt/nifi/test/TemplateProcessor.class
> >
> > The processor works, but selecting "Usage" won't show my HTML file.
> >
> > I understood that I write the HTML file and Nifi will picks it up when it
> > starts. Or is this not true?
> >
> > Thanks for feedback,
> >
> > Uwe
> >
>

Re: Your Distro

[dan bress]

Paul,
   Have you tried joining the mailing list using the instructions here:
https://nifi.apache.org/mailing_lists.html

I know in the past, we have had issues with people who aren't
subscribed being unable to send to the list.  This seems weird to me as
well, but since you are bringing up some concerns and making suggestions it
might be best to join the list so that they can be heard and addressed.

Dan

On Wed, Mar 23, 2016 at 9:41 AM Matthew Clarke 
wrote:

> -- Forwarded message --
> From: Paul Nahay 
> Date: Wed, Mar 23, 2016 at 10:31 AM
> Subject: Your Distro
> To: Matthew Clarke 
>
>
> This is the 2nd time I've gotten an email I sent to
> dev@nifi.apache.org
> returned to me:
>
>
>
> Hi. This is the qmail-send program at apache.org.
> I'm afraid I wasn't able to deliver your message to the following
> addresses.
> This is a permanent error; I've given up. Sorry it didn't work out.
>
> :
> ezmlm-reject: fatal: Sorry, I don't accept messages of MIME
> Content-Type 'text/html' (#5.2.3)
>

Re: @DynamicProperty annotation of more than one property?

[dan bress]

Russell,
   I agree the syntax for multiple DynamicProperties is a little awkward,
and that our documentation could be improved in this situation(I didn't see
reference to @DynamicProperty in the developer doc).

   Also Java8 supports the concept of repeatable annotations[1], which
would erase the need for the @DynamicProperties by allowing you to specify
multiple @DynamicProperty annotations on a single class.  Since NiFi
appears to be running on Java8, I think we should create a ticket to update
the documentation generation code to handle repeated annotations, and
update our processors/controller services/reporting tasks to use them.  We
should also create a ticket to update the developer documentation to
describe the annotations usage.

[1] http://docs.oracle.com/javase/tutorial/java/annotations/repeating.html

Dan

On Tue, Apr 19, 2016 at 7:23 AM Russell Bateman <
russell.bate...@perfectsearchcorp.com> wrote:

> Thanks, Joe. This worked.
>
> You know, I'm just the sort of guy that would add stuff like that to the
> documentation if the way were smoothed to it. The problem as I see it
> with NiFi doc is that it's really good the way it is and if we add
> more--and we should--it's going to get longer and harder to read.
> Someone should slightly redesign it with links to subordinate documents
> to avoid making it too complicated yet give folk a place to go for
> deeper help. I would be happy to help out once such a thing as that is
> decided.
>
> Russ
>
> On 04/15/2016 06:39 PM, Joe Percivall wrote:
> > Hello Russell,
> >
> > The annotation you are looking for is @DynamicProperties[1] an example
> of it in use is in the PutFTP processor[2].
> >
> > [1]
> https://github.com/apache/nifi/blob/e4b7e47836edf47042973e604005058c28eed23b/nifi-api/src/main/java/org/apache/nifi/annotation/behavior/DynamicProperties.java
> > [2]
> https://github.com/apache/nifi/blob/e4b7e47836edf47042973e604005058c28eed23b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutFTP.java#L50
> >
> >
> > Hope that helps,
> > Joe
> >
> > - - - - - -
> > Joseph Percivall
> > linkedin.com/in/Percivall
> > e: joeperciv...@yahoo.com
> >
> >
> >
> > On Friday, April 15, 2016 7:46 PM, Russell Bateman <
> russell.bate...@perfectsearchcorp.com> wrote:
> >
> >
> >
> > What's the syntax for defining more than one dynamic property for a
> > processor? I need to specify up to three distinctly different ones and
> > attempting to do it all in
> >
> > @DynamicProperty(
> >   name = "{blah,blah2,blah3}",
> >   value = "{\"blah-value\",\"blah2-value\",\"blah3-value\"}",
> > supportsExpressionLanguage = false,
> >   description = "Big blah, blah, blah..."
> > )
> >
> > is under-powered.
> >
> > Many thanks.
>
>

Re: [ANNOUNCE] New Apache NiFi Committer James Wing

[dan bress]

Welcome James!

On Wed, May 11, 2016 at 1:15 PM Mark Payne  wrote:

> Welcome, James! Glad to have you aboard!
>
> > On May 11, 2016, at 4:10 PM, Joe Witt  wrote:
> >
> > On behalf of the Apache NiFi PMC, I am pleased to announce that James
> > Wing has accepted the PMC's inviation to become a committer on the
> > Apache NiFi project.  James has been an excellent contributor to a
> > number of code submissions, peer reviews, mailing list discussions and
> > decision making and it is greatly appreciated.  We look forward to his
> > continued involvement in the project.
> >
> > Welcome and congratulations!
>
>

Re: [DISCUSS] - Markdown option for documentation artifacts

[dan bress]

+1

On Wed, Jun 8, 2016 at 7:05 AM Andre  wrote:

> +1 on this + a template that matches existing additional.html
> On 8 Jun 2016 04:28, "Bryan Rosander"  wrote:
>
> > Hey all,
> >
> > When writing documentation (e.g. the additionalDetails.html for a
> > processor) it would be nice to have the option to use Markdown instead of
> > html.
> >
> > I think Markdown is easier to read and write than raw HTML and for simple
> > cases does the job pretty well.  It also has the advantage of being able
> to
> > be translated into other document types easily and it would be rendered
> by
> > default in Github when the file is clicked.
> >
> > There is an MIT-licensed Markdown maven plugin (
> > https://github.com/walokra/markdown-page-generator-plugin) that seems
> like
> > it might work for translating additionalDetails.md (and others) into an
> > equivalent html page.
> >
> > Thanks,
> > Bryan Rosander
> >
>