Re: [VOTE] Release Apache NiFi 1.6.0 (RC2)

[Sivaprasanna]

Thanks, Scott. That helped.

On Thu, 29 Mar 2018 at 10:09 AM, James Wing  wrote:

> +1 (binding).  Ran through the release helper, worked with a test flow.
> Thanks for putting this together.
>
> On Mon, Mar 26, 2018 at 8:34 PM, Joe Witt  wrote:
>
> > Hello,
> >
> > I am pleased to be calling this vote for the source release of Apache
> > NiFi nifi-1.6.0.
> >
> > The source zip, including signatures, digests, etc. can be found at:
> > https://repository.apache.org/content/repositories/orgapachenifi-1123
> >
> > The Git tag is nifi-1.6.0-RC2
> > The Git commit ID is b5935ec81a7cbc048820781ac62cd96bbea5b232
> > https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> > b5935ec81a7cbc048820781ac62cd96bbea5b232
> >
> > Checksums of nifi-1.6.0-source-release.zip:
> > SHA1: 009f1e2e3c17e38f21f27170b9c06228d11653c0
> > SHA256: 39941a5b25427e2b4cc5ba8206084ff92df58863f29ddd097d4ac1e85424beb9
> > SHA512: 1773417a48665e3cda22180ea7f401bc8190ebddbf3f7bc29831e46e7ab0
> > a07694c6e478d252fa573209d4a3c8132a522a8507b6a8784669ab7364847a07e234
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/joewitt.asc
> >
> > KEYS file available here:
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > 146 issues were closed/resolved for this release:
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > projectId=12316020=12342422
> >
> > Release note highlights can be found here:
> > https://cwiki.apache.org/confluence/display/NIFI/
> > Release+Notes#ReleaseNotes-Version1.6.0
> >
> > The vote will be open for 72 hours.
> > Please download the release candidate and evaluate the necessary items
> > including checking hashes, signatures, build
> > from source, and test.  The please vote:
> >
> > [ ] +1 Release this package as nifi-1.6.0
> > [ ] +0 no opinion
> > [ ] -1 Do not release this package because...
> >
>

Re: ListSFTP incoming relationship

[Sivaprasanna]

Should we really have to have an optional state saving functionality? If
the user is unaware of the implications and proceed to store the state then
what Andrew Grande mentioned will happen - possibilities of never ending
stream of state information being stored. If we still go with the optional
state management approach, documentation have to be clear in explaining the
implications.

Sivaprasanna

On Thu, 29 Mar 2018 at 9:28 AM, scott  wrote:

> Okay. So, a new processor called "ScanSFTP", allow incoming relationship
> where the content of the flow file is replaced with the list of matching
> files from the remote directory, then the list is filtered by the usual
> regex parameters like today. Optional state information is kept to
> additionally filter the list of files older than the newest file
> observed during the last run. Does that sound okay to everyone? If so,
> what's the next step?
>
> Scott
>
>
> On 03/27/2018 06:21 PM, scott wrote:
> >
> > This is a great discussion, and appreciate the interest in my problem.
> > I think there are workarounds if you decide not to store state, but
> > I'd recommend keeping it. I think state should be kept optionally,
> > even turned off by default. Several times I've had issues where the
> > state has cause me to miss files, because files get moved into the
> > source folder out of order, and I've wished I could turn the state
> > feature off.
> >
> > In my current use-case, I would not be frequently, dynamically
> > changing the source directory, though I can see the use-cases where it
> > would be. In my current use-case, I want to use an external database
> > table to control the configuration of all my flows. I do this by first
> > reading the content of the table for this particular flow ID, then
> > assign the result as attributes to the flowfile, essentially creating
> > variables I can use throughout the flow to control its behavior. This
> > works great with flows that initiate with HTTP or SQL, but not
> > ListSFTP or ListFile.
> >
> > Scott
> >
> >
> > On 03/27/2018 02:05 PM, Andy LoPresto wrote:
> >> I think Bryan’s point is a good one and when I first saw this
> >> question (and thought of the previous times it’s been asked), my
> >> initial response is to propose a second processor.
> >>
> >> Something like “ScanSFTP”/“IndexSFTP”/“SnapshotSFTP” which operates
> >> differently from ListSFTP — it does not maintain state, and performs
> >> a one-time tabulation/chronicling of the state of that directory at
> >> the given point in time.
> >>
> >> The responsibility to maintain and compare state across time is no
> >> longer a requirement. There could even be a setting in the processor
> >> to allow for “individual flowfile output” (i.e. act the same as
> >> ListSFTP and output one flowfile per item listed) or “summary
> >> flowfile output” where a single flowfile is generated containing the
> >> directory listing information for all the items there. (Another
> >> option is to output both on two different relationships).
> >>
> >> I think this would enable the types of workflows that users have
> >> asked about in the past without compromising the mechanism by which
> >> List* processors work and adding undue complexity to those processors.
> >>
> >> Absolutely crystal clear documentation (and a standard verb for the
> >> new processor family) would be necessary (not only because these
> >> processor solve different problems, but to avoid a million variants
> >> of “I used ScanSFTP processor and it’s not tracking state”/“How do I
> >> provide a directory in an attribute to ListSFTP” mailing list
> >> questions).
> >>
> >>
> >> Andy LoPresto
> >> alopre...@apache.org 
> >> /alopresto.apa...@gmail.com /
> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >>
> >>> On Mar 27, 2018, at 8:33 AM, Andrew Grande  >>> > wrote:
> >>>
> >>> The key here is that ListXXX processor maintains state. A directory
> >>> is part
> >>> of such state. Allowing arbitrary directories via an expression would
> >>> create never ending stream of new entries in the state storage,
> >>> effectively
> >>> engineering a distributed DoS attack on the NiFi node or shared ZK
> >>> quorum
> >>> (for when state is stored in there).
> >>>
> >>> Maybe if we focus on thinking about assumptions and restrictions the
> >>> processor should make to contain that risk...
> >>>
> >>> Andrew
> >>>
> >>> On Tue, Mar 27, 2018, 9:56 AM Bryan Bende  >>> > wrote:
> >>>
>  I'm not sure that would solve the problem because you'd still be
>  limited to one directory. What most people are asking for is the
>  ability to use a dynamic directory from an incoming flow file.
> 
>  I think we might be trying to fit two different use-cases into one
>  processor which might not make sense.
> 

Re: [VOTE] Release Apache NiFi 1.6.0 (RC2)

[James Wing]

+1 (binding).  Ran through the release helper, worked with a test flow.
Thanks for putting this together.

On Mon, Mar 26, 2018 at 8:34 PM, Joe Witt  wrote:

> Hello,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi nifi-1.6.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1123
>
> The Git tag is nifi-1.6.0-RC2
> The Git commit ID is b5935ec81a7cbc048820781ac62cd96bbea5b232
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> b5935ec81a7cbc048820781ac62cd96bbea5b232
>
> Checksums of nifi-1.6.0-source-release.zip:
> SHA1: 009f1e2e3c17e38f21f27170b9c06228d11653c0
> SHA256: 39941a5b25427e2b4cc5ba8206084ff92df58863f29ddd097d4ac1e85424beb9
> SHA512: 1773417a48665e3cda22180ea7f401bc8190ebddbf3f7bc29831e46e7ab0
> a07694c6e478d252fa573209d4a3c8132a522a8507b6a8784669ab7364847a07e234
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 146 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12316020=12342422
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-Version1.6.0
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test.  The please vote:
>
> [ ] +1 Release this package as nifi-1.6.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>

Re: ListSFTP incoming relationship

[scott]

Okay. So, a new processor called "ScanSFTP", allow incoming relationship 
where the content of the flow file is replaced with the list of matching 
files from the remote directory, then the list is filtered by the usual 
regex parameters like today. Optional state information is kept to 
additionally filter the list of files older than the newest file 
observed during the last run. Does that sound okay to everyone? If so, 
what's the next step?


Scott


On 03/27/2018 06:21 PM, scott wrote:


This is a great discussion, and appreciate the interest in my problem. 
I think there are workarounds if you decide not to store state, but 
I'd recommend keeping it. I think state should be kept optionally, 
even turned off by default. Several times I've had issues where the 
state has cause me to miss files, because files get moved into the 
source folder out of order, and I've wished I could turn the state 
feature off.


In my current use-case, I would not be frequently, dynamically 
changing the source directory, though I can see the use-cases where it 
would be. In my current use-case, I want to use an external database 
table to control the configuration of all my flows. I do this by first 
reading the content of the table for this particular flow ID, then 
assign the result as attributes to the flowfile, essentially creating 
variables I can use throughout the flow to control its behavior. This 
works great with flows that initiate with HTTP or SQL, but not 
ListSFTP or ListFile.


Scott


On 03/27/2018 02:05 PM, Andy LoPresto wrote:
I think Bryan’s point is a good one and when I first saw this 
question (and thought of the previous times it’s been asked), my 
initial response is to propose a second processor.


Something like “ScanSFTP”/“IndexSFTP”/“SnapshotSFTP” which operates 
differently from ListSFTP — it does not maintain state, and performs 
a one-time tabulation/chronicling of the state of that directory at 
the given point in time.


The responsibility to maintain and compare state across time is no 
longer a requirement. There could even be a setting in the processor 
to allow for “individual flowfile output” (i.e. act the same as 
ListSFTP and output one flowfile per item listed) or “summary 
flowfile output” where a single flowfile is generated containing the 
directory listing information for all the items there. (Another 
option is to output both on two different relationships).


I think this would enable the types of workflows that users have 
asked about in the past without compromising the mechanism by which 
List* processors work and adding undue complexity to those processors.


Absolutely crystal clear documentation (and a standard verb for the 
new processor family) would be necessary (not only because these 
processor solve different problems, but to avoid a million variants 
of “I used ScanSFTP processor and it’s not tracking state”/“How do I 
provide a directory in an attribute to ListSFTP” mailing list 
questions).



Andy LoPresto
alopre...@apache.org 
/alopresto.apa...@gmail.com /
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Mar 27, 2018, at 8:33 AM, Andrew Grande > wrote:


The key here is that ListXXX processor maintains state. A directory 
is part

of such state. Allowing arbitrary directories via an expression would
create never ending stream of new entries in the state storage, 
effectively
engineering a distributed DoS attack on the NiFi node or shared ZK 
quorum

(for when state is stored in there).

Maybe if we focus on thinking about assumptions and restrictions the
processor should make to contain that risk...

Andrew

On Tue, Mar 27, 2018, 9:56 AM Bryan Bende > wrote:



I'm not sure that would solve the problem because you'd still be
limited to one directory. What most people are asking for is the
ability to use a dynamic directory from an incoming flow file.

I think we might be trying to fit two different use-cases into one
processor which might not make sense.

Scenario #1... There is a directory that is constantly receiving new
data and has a significant amount of files, and I want to periodically
find new files. This is what the current processors are optimized for.

Scenario #2... There is a directory that is mostly static with a
moderate/small number of files, and at points in my flow I want to
dynamically perform a listing of this directory and retrieve the
files. This is more geared towards the mentality of running a
job/workflow.




On Tue, Mar 27, 2018 at 9:36 AM, Otto Fowler 
>

wrote:
What if the changes where ‘on top of’ some base set of properties, 
like

directory?
Like a filter, where if present from the incoming file will have the

LIST*

list only things
that match a name or attribute?



On March 27, 2018 at 

Re: [VOTE] Release Apache NiFi 1.6.0 (RC2)

[Scott Aslan]

+1 (binding)

- Ran through release helper
- Setup secure NiFi and verified a test flow

Sivaprasanna, I believe the issue you are seeing is due to npm not playing
well with your linux distro. If you checkout latest master it contains this
PR  which should allow you to
complete your build.

-Scott

On Wed, Mar 28, 2018 at 2:01 PM, Mark Payne  wrote:

> +1 (binding)
>
> Was able to build successfully and start application. Created some simple
> flows to verify expected behavior.
> Verified hashes and signing.
>
> Thanks
> -Mark
>
>
>
> > On Mar 26, 2018, at 11:34 PM, Joe Witt  wrote:
> >
> > Hello,
> >
> > I am pleased to be calling this vote for the source release of Apache
> > NiFi nifi-1.6.0.
> >
> > The source zip, including signatures, digests, etc. can be found at:
> > https://repository.apache.org/content/repositories/orgapachenifi-1123
> >
> > The Git tag is nifi-1.6.0-RC2
> > The Git commit ID is b5935ec81a7cbc048820781ac62cd96bbea5b232
> > https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> b5935ec81a7cbc048820781ac62cd96bbea5b232
> >
> > Checksums of nifi-1.6.0-source-release.zip:
> > SHA1: 009f1e2e3c17e38f21f27170b9c06228d11653c0
> > SHA256: 39941a5b25427e2b4cc5ba8206084ff92df58863f29ddd097d4ac1e85424beb9
> > SHA512: 1773417a48665e3cda22180ea7f401bc8190ebddbf3f7bc29831e46e7ab0
> a07694c6e478d252fa573209d4a3c8132a522a8507b6a8784669ab7364847a07e234
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/joewitt.asc
> >
> > KEYS file available here:
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > 146 issues were closed/resolved for this release:
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12316020=12342422
> >
> > Release note highlights can be found here:
> > https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-Version1.6.0
> >
> > The vote will be open for 72 hours.
> > Please download the release candidate and evaluate the necessary items
> > including checking hashes, signatures, build
> > from source, and test.  The please vote:
> >
> > [ ] +1 Release this package as nifi-1.6.0
> > [ ] +0 no opinion
> > [ ] -1 Do not release this package because...
>
>

Re: Anything like this been implemented yet?

[Andy LoPresto]

I would also suggest if you want to set a static flow and not have to update 
data directly in NiFi, put the JSON strings in a file and use GetFile to read 
it in, SplitText to split it into individual lines (flowfiles), and process 
that way.


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Mar 28, 2018, at 2:13 PM, Bryan Bende  wrote:
> 
> Since it sounds like each query is a JSON document, can you create a
> JSON array of all your queries and put that as the Custom Text of a
> GenerateFlowFile processor?
> 
> Then follow it with SplitJson to split the array into a query per flow
> file, assuming that is what you want.
> 
> Could also use ExecuteScript and enter all the queries as user defined
> properties, then write a small script that loops of the properties and
> produces a flow file for each dynamic property, where the content is
> the value of the property which would be the query string.
> 
> On Wed, Mar 28, 2018 at 5:03 PM, Mike Thomsen  wrote:
>> More specifically: I know you can do this functionality by chaining other
>> processors or using a bunch of generateflowfiles. What I'm looking for is a
>> one stop way of associating dozens of small queries to one processor and
>> have it batch send them on its own with no backend dependency like a
>> database.
>> 
>> On Wed, Mar 28, 2018 at 5:02 PM, Mike Thomsen 
>> wrote:
>> 
>>> I have a client that would benefit from being able to run certain queries
>>> periodically. Like half a dozen or more. Is there any processor where you
>>> can associate a bunch of strings (like JSON) and send them out individually
>>> as flowfiles?
>>> 
>>> Thanks,
>>> 
>>> Mike
>>> 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Anything like this been implemented yet?

[Bryan Bende]

Since it sounds like each query is a JSON document, can you create a
JSON array of all your queries and put that as the Custom Text of a
GenerateFlowFile processor?

Then follow it with SplitJson to split the array into a query per flow
file, assuming that is what you want.

Could also use ExecuteScript and enter all the queries as user defined
properties, then write a small script that loops of the properties and
produces a flow file for each dynamic property, where the content is
the value of the property which would be the query string.

On Wed, Mar 28, 2018 at 5:03 PM, Mike Thomsen  wrote:
> More specifically: I know you can do this functionality by chaining other
> processors or using a bunch of generateflowfiles. What I'm looking for is a
> one stop way of associating dozens of small queries to one processor and
> have it batch send them on its own with no backend dependency like a
> database.
>
> On Wed, Mar 28, 2018 at 5:02 PM, Mike Thomsen 
> wrote:
>
>> I have a client that would benefit from being able to run certain queries
>> periodically. Like half a dozen or more. Is there any processor where you
>> can associate a bunch of strings (like JSON) and send them out individually
>> as flowfiles?
>>
>> Thanks,
>>
>> Mike
>>

Re: Anything like this been implemented yet?

[Mike Thomsen]

More specifically: I know you can do this functionality by chaining other
processors or using a bunch of generateflowfiles. What I'm looking for is a
one stop way of associating dozens of small queries to one processor and
have it batch send them on its own with no backend dependency like a
database.

On Wed, Mar 28, 2018 at 5:02 PM, Mike Thomsen 
wrote:

> I have a client that would benefit from being able to run certain queries
> periodically. Like half a dozen or more. Is there any processor where you
> can associate a bunch of strings (like JSON) and send them out individually
> as flowfiles?
>
> Thanks,
>
> Mike
>

Anything like this been implemented yet?

[Mike Thomsen]

I have a client that would benefit from being able to run certain queries
periodically. Like half a dozen or more. Is there any processor where you
can associate a bunch of strings (like JSON) and send them out individually
as flowfiles?

Thanks,

Mike

Re: [VOTE] Release Apache NiFi 1.6.0 (RC2)

[Michael Moser]

+1 (binding)

Followed helper to verify release, built on Ubuntu 16.04 LTS, ran the
application successfully.

-- Mike


On Wed, Mar 28, 2018 at 2:01 PM, Mark Payne  wrote:

> +1 (binding)
>
> Was able to build successfully and start application. Created some simple
> flows to verify expected behavior.
> Verified hashes and signing.
>
> Thanks
> -Mark
>
>
>
> > On Mar 26, 2018, at 11:34 PM, Joe Witt  wrote:
> >
> > Hello,
> >
> > I am pleased to be calling this vote for the source release of Apache
> > NiFi nifi-1.6.0.
> >
> > The source zip, including signatures, digests, etc. can be found at:
> > https://repository.apache.org/content/repositories/orgapachenifi-1123
> >
> > The Git tag is nifi-1.6.0-RC2
> > The Git commit ID is b5935ec81a7cbc048820781ac62cd96bbea5b232
> > https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> b5935ec81a7cbc048820781ac62cd96bbea5b232
> >
> > Checksums of nifi-1.6.0-source-release.zip:
> > SHA1: 009f1e2e3c17e38f21f27170b9c06228d11653c0
> > SHA256: 39941a5b25427e2b4cc5ba8206084ff92df58863f29ddd097d4ac1e85424beb9
> > SHA512: 1773417a48665e3cda22180ea7f401bc8190ebddbf3f7bc29831e46e7ab0
> a07694c6e478d252fa573209d4a3c8132a522a8507b6a8784669ab7364847a07e234
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/joewitt.asc
> >
> > KEYS file available here:
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > 146 issues were closed/resolved for this release:
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12316020=12342422
> >
> > Release note highlights can be found here:
> > https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-Version1.6.0
> >
> > The vote will be open for 72 hours.
> > Please download the release candidate and evaluate the necessary items
> > including checking hashes, signatures, build
> > from source, and test.  The please vote:
> >
> > [ ] +1 Release this package as nifi-1.6.0
> > [ ] +0 no opinion
> > [ ] -1 Do not release this package because...
>
>

Re: Docker NiFi Container superuser password

[Joseph Niemiec]

please ignore, i managed to do this by telling docker to login as root and
not the default...

*docker exec -i -t --user root 39dbe311249e bash *

On Wed, Mar 28, 2018 at 2:06 PM, Joseph Niemiec 
wrote:

> Hi everyone.
>
> Looking to change the nameservers in the default docker image
> apache/nifi:latest but am unsure of the password...
>
> nifi@39dbe311249e:/opt/nifi/nifi-1.5.0$ su
> Password:   su: Authentication failure
>
> Is this something that is publicly available?
>
>
> --
> Joseph
>



-- 
Joseph

Docker NiFi Container superuser password

[Joseph Niemiec]

Hi everyone.

Looking to change the nameservers in the default docker image
apache/nifi:latest but am unsure of the password...

nifi@39dbe311249e:/opt/nifi/nifi-1.5.0$ su
Password:  

Re: [VOTE] Release Apache NiFi 1.6.0 (RC2)

[Mark Payne]

+1 (binding)

Was able to build successfully and start application. Created some simple flows 
to verify expected behavior.
Verified hashes and signing.

Thanks
-Mark



> On Mar 26, 2018, at 11:34 PM, Joe Witt  wrote:
> 
> Hello,
> 
> I am pleased to be calling this vote for the source release of Apache
> NiFi nifi-1.6.0.
> 
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1123
> 
> The Git tag is nifi-1.6.0-RC2
> The Git commit ID is b5935ec81a7cbc048820781ac62cd96bbea5b232
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=b5935ec81a7cbc048820781ac62cd96bbea5b232
> 
> Checksums of nifi-1.6.0-source-release.zip:
> SHA1: 009f1e2e3c17e38f21f27170b9c06228d11653c0
> SHA256: 39941a5b25427e2b4cc5ba8206084ff92df58863f29ddd097d4ac1e85424beb9
> SHA512: 
> 1773417a48665e3cda22180ea7f401bc8190ebddbf3f7bc29831e46e7ab0a07694c6e478d252fa573209d4a3c8132a522a8507b6a8784669ab7364847a07e234
> 
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
> 
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
> 
> 146 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12342422
> 
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.6.0
> 
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test.  The please vote:
> 
> [ ] +1 Release this package as nifi-1.6.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...

Re: [VOTE] Release Apache NiFi 1.6.0 (RC2)

[Sivaprasanna]

- Confirmed hashes and signature

Tried to build in a fresh Ubuntu VM that doesn't have NiFi installed or
configured previously. Build fails at nifi-web-ui module. Going by how it
works for others, it very well could be a isolated problem on my end.
However still wanted to get this posted.

Error snippet:

[ERROR] npm ERR! Error: No dist in undefined package
[ERROR] npm ERR! at next
(/opt/tmp/nifi-1.6.0/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node_modules/npm/lib/cache.js:746:26)
[ERROR] npm ERR! at
/opt/tmp/nifi-1.6.0/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node_modules/npm/lib/cache.js:739:5
[ERROR] npm ERR! at RegClient.get_
(/opt/tmp/nifi-1.6.0/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node_modules/npm/node_modules/npm-registry-client/lib/get.js:105:14)

Also attached the error message.

Environment:
Ubuntu 16.04 LTS x64

Thanks,
Sivaprasanna

On Wed, Mar 28, 2018 at 8:31 PM, Kevin Doran  wrote:

> +1
>
> I ran through the steps in the release helper guide and everything looks
> good.
>
> Thanks for handling RM duties, Joe.
>
> --Kevin
>
> On 3/26/18, 23:34, "Joe Witt"  wrote:
>
> Hello,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi nifi-1.6.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1123
>
> The Git tag is nifi-1.6.0-RC2
> The Git commit ID is b5935ec81a7cbc048820781ac62cd96bbea5b232
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> b5935ec81a7cbc048820781ac62cd96bbea5b232
>
> Checksums of nifi-1.6.0-source-release.zip:
> SHA1: 009f1e2e3c17e38f21f27170b9c06228d11653c0
> SHA256: 39941a5b25427e2b4cc5ba8206084ff92df58863f29ddd097d4ac1e85424
> beb9
> SHA512: 1773417a48665e3cda22180ea7f401bc8190ebddbf3f7bc29831e46e7ab0
> a07694c6e478d252fa573209d4a3c8132a522a8507b6a8784669ab7364847a07e234
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 146 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12316020=12342422
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-Version1.6.0
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test.  The please vote:
>
> [ ] +1 Release this package as nifi-1.6.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>
>
>
>
INFO] Installed node locally.
[INFO] Installing npm version 1.3.8
[INFO] Unpacking 
/root/.m2/repository/com/github/eirslett/npm/1.3.8/npm-1.3.8.tar.gz into 
/opt/tmp/nifi-1.6.0/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node_modules
[INFO] Installed npm locally.
[INFO] 
[INFO] --- frontend-maven-plugin:1.1:npm (npm install) @ nifi-web-ui ---
[DEBUG] Configuring mojo com.github.eirslett:frontend-maven-plugin:1.1:npm from 
plugin realm ClassRealm[plugin>com.github.eirslett:frontend-maven-plugin:1.1, 
parent: sun.misc.Launcher$AppClassLoader@33909752]
[DEBUG] Configuring mojo 'com.github.eirslett:frontend-maven-plugin:1.1:npm' 
with basic configurator -->
[DEBUG]   (f) arguments = --cache-min Infinity install
[DEBUG]   (f) installDirectory = 
/opt/tmp/nifi-1.6.0/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory
[DEBUG]   (f) npmInheritsProxyConfigFromMaven = true
[DEBUG]   (f) project = MavenProject: org.apache.nifi:nifi-web-ui:1.6.0 @ 
/opt/tmp/nifi-1.6.0/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/pom.xml
[DEBUG]   (f) repositorySystemSession = 
org.eclipse.aether.DefaultRepositorySystemSession@3b0d3a63
[DEBUG]   (f) session = org.apache.maven.execution.MavenSession@5568c66f
[DEBUG]   (f) skip = false
[DEBUG]   (f) skipTests = false
[DEBUG]   (f) workingDirectory = 
/opt/tmp/nifi-1.6.0/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory
[DEBUG]   (f) execution = com.github.eirslett:frontend-maven-plugin:1.1:npm 
{execution: npm install}
[DEBUG] -- end configuration --
[INFO] Running 'npm --cache-min Infinity install' in 
/opt/tmp/nifi-1.6.0/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory
[WARNING] npm WARN package.json apache-nifi@ No 

Re: A bug in expression substitution

[Pierre Villard]

Probably related to this JIRA:
https://issues.apache.org/jira/browse/NIFI-4407

2018-03-28 18:42 GMT+02:00 Mark Payne :

> Sergei,
>
> Thanks for reporting this! I have created a JIRA [1] to track this.
>
> -Mark
>
> [1] https://issues.apache.org/jira/browse/NIFI-5026
>
>
> On Mar 28, 2018, at 7:36 AM, Sergei Zhirikov  mailto:sf...@yahoo.com.INVALID>> wrote:
>
> Hi,
> It looks like I have stumbled upon a bug in substitution of evaluated
> expressions.
> A test case:
> String result = org.apache.nifi.attribute.expression.language.Query.
> prepare("${foo}$${foo}").evaluateExpressions(
> Collections.singletonMap("foo", "bar"), null);
> Expected result: "bar${foo}"Observed result: "barbar"
> The issue exists in 1.5.0 and, as far as I can tell, in the master
> branch.The cause is quite simple: Query.prepare(...) splits the input
> string into pieces to be evaluated and substituted or to be copied
> literally, but it doesn't keep track of which is which.That couldn't
> possibly work. If a piece to be copied literally happens to be equal to one
> of the pieces to be substituted, the things go wrong, as the test case
> demonstrates.
>
> Regards,Sergei.
>
>
>

Re: A bug in expression substitution

[Mark Payne]

Sergei,

Thanks for reporting this! I have created a JIRA [1] to track this.

-Mark

[1] https://issues.apache.org/jira/browse/NIFI-5026


On Mar 28, 2018, at 7:36 AM, Sergei Zhirikov 
> wrote:

Hi,
It looks like I have stumbled upon a bug in substitution of evaluated 
expressions.
A test case:
String result = 
org.apache.nifi.attribute.expression.language.Query.prepare("${foo}$${foo}").evaluateExpressions(Collections.singletonMap("foo",
 "bar"), null);
Expected result: "bar${foo}"Observed result: "barbar"
The issue exists in 1.5.0 and, as far as I can tell, in the master branch.The 
cause is quite simple: Query.prepare(...) splits the input string into pieces 
to be evaluated and substituted or to be copied literally, but it doesn't keep 
track of which is which.That couldn't possibly work. If a piece to be copied 
literally happens to be equal to one of the pieces to be substituted, the 
things go wrong, as the test case demonstrates.

Regards,Sergei.

Re: ForkJoinPool.commonPool() in Nifi

[Pierre Villard]

Hi Oleksi,

I'm not familiar with this part of the code but raising a JIRA sounds valid
to me.
If there is no fix for it, at least that is useful information and it could
help other people seeing the same behavior.

Pierre

2018-03-28 15:47 GMT+02:00 Otto Fowler :

> I would think NiFi should have it’s own thread pool.
>
>
> On March 28, 2018 at 09:29:31, Oleksi Derkatch (oderka...@vitalimages.com)
> wrote:
>
> Anyone have any thoughts on this? Should I make a JIRA ticket?
>
> 
> From: Oleksi Derkatch 
> Sent: Thursday, March 8, 2018 4:36:51 PM
> To: dev@nifi.apache.org
> Subject: ForkJoinPool.commonPool() in Nifi
>
> Hi,
>
>
> A few weeks ago we encountered a problem with one of our custom processors
> which seemed to deadlock all processing in Nifi under load. We believe the
> issue is that our processors were relying on ForkJoinPool.commonPool, but
> so was the Nifi engine during it's scheduling (both via CompletableFuture).
> As such, when we did a thread dump, we saw something like this:
>
>
> "ForkJoinPool.commonPool-worker-6" #381 daemon prio=5 os_prio=0
> tid=0x7f300d934000 nid=0x4be4 waiting on condition [0x7f2fd53e7000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0006c8b00568> (a
> java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.CompletableFuture$Signaller.
> block(CompletableFuture.java:1693)
>
> at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3313)
> at
> java.util.concurrent.CompletableFuture.waitingGet(
> CompletableFuture.java:1729)
>
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at customcode(customcode.java:83)
> at customcode.lambda$null$23(customcode.java:320)
> at customcode$$Lambda$261/442205945.call(Unknown Source)
> at
> com.google.common.cache.LocalCache$LocalManualCache$1.
> load(LocalCache.java:4724)
>
> at
> com.google.common.cache.LocalCache$LoadingValueReference.
> loadFuture(LocalCache.java:3522)
>
> at
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2315)
> at
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.
> java:2278)
>
> - locked <0x0006c8b007f0> (a
> com.google.common.cache.LocalCache$StrongWriteEntry)
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2193)
> at com.google.common.cache.LocalCache.get(LocalCache.java:3932)
> at
> com.google.common.cache.LocalCache$LocalManualCache.
> get(LocalCache.java:4721)
>
> at customcode.lambda$customethod$24(customcode.java:309)
> at customcode$$Lambda$258/1540137328.get(Unknown Source)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.
> run(CompletableFuture.java:1590)
>
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.
> exec(CompletableFuture.java:1582)
>
> at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:
> 1056)
> at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> at
> java.util.concurrent.ForkJoinWorkerThread.run(
> ForkJoinWorkerThread.java:157)
>
>
>
> I think what happened here is that since we were both using
> ForkJoinPool.commonPool() like this, we actually ran out of threads and
> deadlocked. We were waiting (in the nifi processor) on a future that was
> also submitted to the same commonPool at the time of the deadlock. The
> solution was for us to use a dedicated thread pool instead of a shared one.
>
>
> It might be worth considering changing this in Nifi for the future, in case
> other custom processors use this pattern as well.
>
>
> This also brings up another question. By default, the size of this thread
> pool is (# of CPUs - 1). How does that affect processing when we set the
> maximum number of threads in the Nifi UI to be much higher than that?
> Shouldn't this thread pool be configured for the same size? This is tunable
> with the -Djava.util.concurrent.ForkJoinPool.common.parallelism java flag
> (which we also adjusted in troubleshooting this).
>
>
>
>
>
>
>
>
>
>
> Notice - Confidential Information The information in this communication and
> any attachments is strictly confidential and intended only for the use of
> the individual(s) or entity(ies) named above. If you are not the intended
> recipient, any dissemination, distribution, copying or other use of the
> information contained in this communication and/or any attachment is
> strictly prohibited. If you have received this communication in error,
> please first notify the sender immediately and then delete this
> communication from all data storage devices and destroy all hard copies.
>

Re: [VOTE] Release Apache NiFi 1.6.0 (RC2)

[Kevin Doran]

+1

I ran through the steps in the release helper guide and everything looks good.

Thanks for handling RM duties, Joe.

--Kevin

On 3/26/18, 23:34, "Joe Witt"  wrote:

Hello,

I am pleased to be calling this vote for the source release of Apache
NiFi nifi-1.6.0.

The source zip, including signatures, digests, etc. can be found at:
https://repository.apache.org/content/repositories/orgapachenifi-1123

The Git tag is nifi-1.6.0-RC2
The Git commit ID is b5935ec81a7cbc048820781ac62cd96bbea5b232

https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=b5935ec81a7cbc048820781ac62cd96bbea5b232

Checksums of nifi-1.6.0-source-release.zip:
SHA1: 009f1e2e3c17e38f21f27170b9c06228d11653c0
SHA256: 39941a5b25427e2b4cc5ba8206084ff92df58863f29ddd097d4ac1e85424beb9
SHA512: 
1773417a48665e3cda22180ea7f401bc8190ebddbf3f7bc29831e46e7ab0a07694c6e478d252fa573209d4a3c8132a522a8507b6a8784669ab7364847a07e234

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/joewitt.asc

KEYS file available here:
https://dist.apache.org/repos/dist/release/nifi/KEYS

146 issues were closed/resolved for this release:

https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12342422

Release note highlights can be found here:

https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.6.0

The vote will be open for 72 hours.
Please download the release candidate and evaluate the necessary items
including checking hashes, signatures, build
from source, and test.  The please vote:

[ ] +1 Release this package as nifi-1.6.0
[ ] +0 no opinion
[ ] -1 Do not release this package because...

Re: ForkJoinPool.commonPool() in Nifi

[Otto Fowler]

I would think NiFi should have it’s own thread pool.


On March 28, 2018 at 09:29:31, Oleksi Derkatch (oderka...@vitalimages.com)
wrote:

Anyone have any thoughts on this? Should I make a JIRA ticket?


From: Oleksi Derkatch 
Sent: Thursday, March 8, 2018 4:36:51 PM
To: dev@nifi.apache.org
Subject: ForkJoinPool.commonPool() in Nifi

Hi,


A few weeks ago we encountered a problem with one of our custom processors
which seemed to deadlock all processing in Nifi under load. We believe the
issue is that our processors were relying on ForkJoinPool.commonPool, but
so was the Nifi engine during it's scheduling (both via CompletableFuture).
As such, when we did a thread dump, we saw something like this:


"ForkJoinPool.commonPool-worker-6" #381 daemon prio=5 os_prio=0
tid=0x7f300d934000 nid=0x4be4 waiting on condition [0x7f2fd53e7000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0006c8b00568> (a
java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1693)

at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3313)
at
java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1729)

at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at customcode(customcode.java:83)
at customcode.lambda$null$23(customcode.java:320)
at customcode$$Lambda$261/442205945.call(Unknown Source)
at
com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4724)

at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3522)

at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2315)
at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2278)

- locked <0x0006c8b007f0> (a
com.google.common.cache.LocalCache$StrongWriteEntry)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2193)
at com.google.common.cache.LocalCache.get(LocalCache.java:3932)
at
com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4721)

at customcode.lambda$customethod$24(customcode.java:309)
at customcode$$Lambda$258/1540137328.get(Unknown Source)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)

at
java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1582)

at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at
java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)



I think what happened here is that since we were both using
ForkJoinPool.commonPool() like this, we actually ran out of threads and
deadlocked. We were waiting (in the nifi processor) on a future that was
also submitted to the same commonPool at the time of the deadlock. The
solution was for us to use a dedicated thread pool instead of a shared one.


It might be worth considering changing this in Nifi for the future, in case
other custom processors use this pattern as well.


This also brings up another question. By default, the size of this thread
pool is (# of CPUs - 1). How does that affect processing when we set the
maximum number of threads in the Nifi UI to be much higher than that?
Shouldn't this thread pool be configured for the same size? This is tunable
with the -Djava.util.concurrent.ForkJoinPool.common.parallelism java flag
(which we also adjusted in troubleshooting this).










Notice - Confidential Information The information in this communication and
any attachments is strictly confidential and intended only for the use of
the individual(s) or entity(ies) named above. If you are not the intended
recipient, any dissemination, distribution, copying or other use of the
information contained in this communication and/or any attachment is
strictly prohibited. If you have received this communication in error,
please first notify the sender immediately and then delete this
communication from all data storage devices and destroy all hard copies.

Re: ForkJoinPool.commonPool() in Nifi

[Oleksi Derkatch]

Anyone have any thoughts on this? Should I make a JIRA ticket?


From: Oleksi Derkatch 
Sent: Thursday, March 8, 2018 4:36:51 PM
To: dev@nifi.apache.org
Subject: ForkJoinPool.commonPool() in Nifi

Hi,


A few weeks ago we encountered a problem with one of our custom processors 
which seemed to deadlock all processing in Nifi under load. We believe the 
issue is that our processors were relying on ForkJoinPool.commonPool, but so 
was the Nifi engine during it's scheduling (both via CompletableFuture). As 
such, when we did a thread dump, we saw something like this:


"ForkJoinPool.commonPool-worker-6" #381 daemon prio=5 os_prio=0 
tid=0x7f300d934000 nid=0x4be4 waiting on condition [0x7f2fd53e7000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0006c8b00568> (a 
java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at 
java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1693)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3313)
at 
java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1729)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at customcode(customcode.java:83)
at customcode.lambda$null$23(customcode.java:320)
at customcode$$Lambda$261/442205945.call(Unknown Source)
at 
com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4724)
at 
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3522)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2315)
at 
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2278)
- locked <0x0006c8b007f0> (a 
com.google.common.cache.LocalCache$StrongWriteEntry)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2193)
at com.google.common.cache.LocalCache.get(LocalCache.java:3932)
at 
com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4721)
at customcode.lambda$customethod$24(customcode.java:309)
at customcode$$Lambda$258/1540137328.get(Unknown Source)
at 
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
at 
java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1582)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at 
java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at 
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)


I think what happened here is that since we were both using 
ForkJoinPool.commonPool() like this, we actually ran out of threads and 
deadlocked. We were waiting (in the nifi processor) on a future that was also 
submitted to the same commonPool at the time of the deadlock. The solution was 
for us to use a dedicated thread pool instead of a shared one.


It might be worth considering changing this in Nifi for the future, in case 
other custom processors use this pattern as well.


This also brings up another question. By default, the size of this thread pool 
is (# of CPUs - 1). How does that affect processing when we set the maximum 
number of threads in the Nifi UI to be much higher than that? Shouldn't this 
thread pool be configured for the same size? This is tunable with the 
-Djava.util.concurrent.ForkJoinPool.common.parallelism java flag (which we also 
adjusted in troubleshooting this).










Notice - Confidential Information The information in this communication and any 
attachments is strictly confidential and intended only for the use of the 
individual(s) or entity(ies) named above. If you are not the intended 
recipient, any dissemination, distribution, copying or other use of the 
information contained in this communication and/or any attachment is strictly 
prohibited. If you have received this communication in error, please first 
notify the sender immediately and then delete this communication from all data 
storage devices and destroy all hard copies.

Re: Read processor property in init()

[Sivaprasanna]

Just to add on top of what Mike said. The @OnScheduled annotation indicates
that the method that is marked with this annotation will run when a
processor is started every time. So basically the setup() will be called
and executed everytime the processor is started from the UI.

On Wed, 28 Mar 2018 at 6:34 PM, Jeff Zemerick  wrote:

> I will give that a go. Thanks for the quick answer, Mike!
>
> On Wed, Mar 28, 2018 at 9:01 AM, Mike Thomsen 
> wrote:
> > Just do...
> >
> > @OnScheduled
> > public void setup(ProcessContext context) {
> > //Read properties and do setup.
> > }
> >
> > On Wed, Mar 28, 2018 at 8:57 AM, Jeff Zemerick 
> wrote:
> >
> >> Hi everyone,
> >>
> >> Is there a recommended method for making user-configurable property
> >> values available to a processor's init()? I would like to load a large
> >> index file but allow the user to specify the index's path. I am
> >> guessing that init() is executed too early to read user properties.
> >>
> >> Thanks for any suggestions.
> >>
> >> Jeff
> >>
>

Re: Read processor property in init()

[Jeff Zemerick]

I will give that a go. Thanks for the quick answer, Mike!

On Wed, Mar 28, 2018 at 9:01 AM, Mike Thomsen  wrote:
> Just do...
>
> @OnScheduled
> public void setup(ProcessContext context) {
> //Read properties and do setup.
> }
>
> On Wed, Mar 28, 2018 at 8:57 AM, Jeff Zemerick  wrote:
>
>> Hi everyone,
>>
>> Is there a recommended method for making user-configurable property
>> values available to a processor's init()? I would like to load a large
>> index file but allow the user to specify the index's path. I am
>> guessing that init() is executed too early to read user properties.
>>
>> Thanks for any suggestions.
>>
>> Jeff
>>

Re: Read processor property in init()

[Mike Thomsen]

Just do...

@OnScheduled
public void setup(ProcessContext context) {
//Read properties and do setup.
}

On Wed, Mar 28, 2018 at 8:57 AM, Jeff Zemerick  wrote:

> Hi everyone,
>
> Is there a recommended method for making user-configurable property
> values available to a processor's init()? I would like to load a large
> index file but allow the user to specify the index's path. I am
> guessing that init() is executed too early to read user properties.
>
> Thanks for any suggestions.
>
> Jeff
>

Read processor property in init()

[Jeff Zemerick]

Hi everyone,

Is there a recommended method for making user-configurable property
values available to a processor's init()? I would like to load a large
index file but allow the user to specify the index's path. I am
guessing that init() is executed too early to read user properties.

Thanks for any suggestions.

Jeff

Re: [VOTE] Release Apache NiFi 1.6.0 (RC2)

[Sivaprasanna]

Don't have my Linux machine on hand now so tried building on Windows 7 x64
and it fails.

[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.132 s - in org.apache.nifi.wali.TestHashMapSnapshot
[INFO] Running org.apache.nifi.wali.TestLengthDelimitedJournal
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.056 s - in org.apache.nifi.wali.TestLengthDelimitedJournal
[INFO] Running org.apache.nifi.wali.TestSequentialAccessWriteAheadLog
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
0.184 s - in org.apache.nifi.wali.TestSequentialAccessWriteAheadLog
[INFO] Running org.wali.TestMinimalLockingWriteAheadLog
[ERROR] Tests run: 12, Failures: 0, Errors: 1, Skipped: 2, Time elapsed:
12.398 s <<< FAILURE! - in org.wali.TestMinimalLockingWriteAheadLog
[ERROR]
testRecoverFileThatHasTrailingNULBytesAndTruncation(org.wali.TestMinimalLockingWriteAheadLog)
Time elapsed: 0.032 s  <<< ERROR!
java.nio.channels.OverlappingFileLockException
at
org.wali.TestMinimalLockingWriteAheadLog.testRecoverFileThatHasTrailingNULBytesAndTruncation(TestMinimalLockingWriteAheadLog.java:503)

[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]
TestMinimalLockingWriteAheadLog.testRecoverFileThatHasTrailingNULBytesAndTruncation:503
» OverlappingFileLock
[INFO]
[ERROR] Tests run: 30, Failures: 0, Errors: 1, Skipped: 3
[INFO]
[INFO]


Known issue with building on Windows? I have attached the surefire reports.
I'll test the same on a Ubuntu setup and update.

-
Sivaprasanna

On Wed, Mar 28, 2018 at 4:21 PM, Marc Parisi  wrote:

> +1 binding
>
> Built and tested flow on osx and centos
> Worked through release helper's guide
> Validates Sig's and hashes.
>
>
>
> On Wed, Mar 28, 2018, 6:10 AM Pierre Villard 
> wrote:
>
> > +1 (binding)
> >
> > - went through the release helper guide
> > - full clean install on OS X
> > - test unsecured and secured cluster setups and communications with
> > Registry and MiNiFi Java
> > - various flows I have are running as expected
> > - confirmed the fix that blocked the previous RC
> >
> > Thanks Joe for taking care of the RM duties and thanks to everyone that
> > contributed in this release.
> >
> > Pierre
> >
> > 2018-03-28 11:46 GMT+02:00 Koji Kawamura :
> >
> > > +1 (binding)
> > >
> > > - Confirmed hashes
> > > - Built with include-atlas profile
> > > - Confirmed various flows with 3 node secured cluster on Ubuntu
> > > - Tested integration with Hadoop environment and NiFi Registry
> > >
> > > Koji
> > >
> > > On Wed, Mar 28, 2018 at 12:27 PM, Andrew Lim <
> andrewlim.apa...@gmail.com
> > >
> > > wrote:
> > > > +1 (non-binding)
> > > >
> > > > -Ran full clean install on OS X (10.11.6)
> > > > -Tested integration with Secure NiFi Registry (1.5.0)
> > > > -Tested fine grained restricted component policies.  Verified two
> > issues
> > > discovered while testing RC1 have been fixed in RC2 [1, 2]
> > > > -Ran basic flows successfully
> > > > -Reviewed documentation
> > > >
> > > > Drew
> > > >
> > > > [1] https://issues.apache.org/jira/browse/NIFI-5008
> > > > [2] https://issues.apache.org/jira/browse/NIFI-5009
> > > >
> > > >
> > > >> On Mar 26, 2018, at 11:34 PM, Joe Witt  wrote:
> > > >>
> > > >> Hello,
> > > >>
> > > >> I am pleased to be calling this vote for the source release of
> Apache
> > > >> NiFi nifi-1.6.0.
> > > >>
> > > >> The source zip, including signatures, digests, etc. can be found at:
> > > >> https://repository.apache.org/content/repositories/
> orgapachenifi-1123
> > > >>
> > > >> The Git tag is nifi-1.6.0-RC2
> > > >> The Git commit ID is b5935ec81a7cbc048820781ac62cd96bbea5b232
> > > >> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> > > b5935ec81a7cbc048820781ac62cd96bbea5b232
> > > >>
> > > >> Checksums of nifi-1.6.0-source-release.zip:
> > > >> SHA1: 009f1e2e3c17e38f21f27170b9c06228d11653c0
> > > >> SHA256: 39941a5b25427e2b4cc5ba8206084f
> f92df58863f29ddd097d4ac1e85424
> > > beb9
> > > >> SHA512: 1773417a48665e3cda22180ea7f401
> bc8190ebddbf3f7bc29831e46e7ab0
> > > a07694c6e478d252fa573209d4a3c8132a522a8507b6a8784669ab7364847a07e234
> > > >>
> > > >> Release artifacts are signed with the following key:
> > > >> https://people.apache.org/keys/committer/joewitt.asc
> > > >>
> > > >> KEYS file available here:
> > > >> https://dist.apache.org/repos/dist/release/nifi/KEYS
> > > >>
> > > >> 146 issues were closed/resolved for this release:
> > > >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > > projectId=12316020=12342422
> > > >>
> > > >> Release note highlights can be found here:
> > > >> https://cwiki.apache.org/confluence/display/NIFI/
> > > Release+Notes#ReleaseNotes-Version1.6.0
> > > >>
> > > >> The vote will be open for 72 hours.
> > > >> Please download the release candidate and evaluate the necessary
> items
> 

A bug in expression substitution

[Sergei Zhirikov]

Hi,
It looks like I have stumbled upon a bug in substitution of evaluated 
expressions.
A test case:
String result = 
org.apache.nifi.attribute.expression.language.Query.prepare("${foo}$${foo}").evaluateExpressions(Collections.singletonMap("foo",
 "bar"), null);
Expected result: "bar${foo}"Observed result: "barbar"
The issue exists in 1.5.0 and, as far as I can tell, in the master branch.The 
cause is quite simple: Query.prepare(...) splits the input string into pieces 
to be evaluated and substituted or to be copied literally, but it doesn't keep 
track of which is which.That couldn't possibly work. If a piece to be copied 
literally happens to be equal to one of the pieces to be substituted, the 
things go wrong, as the test case demonstrates.

Regards,Sergei.

Re: [VOTE] Release Apache NiFi 1.6.0 (RC2)

[Marc Parisi]

+1 binding

Built and tested flow on osx and centos
Worked through release helper's guide
Validates Sig's and hashes.



On Wed, Mar 28, 2018, 6:10 AM Pierre Villard 
wrote:

> +1 (binding)
>
> - went through the release helper guide
> - full clean install on OS X
> - test unsecured and secured cluster setups and communications with
> Registry and MiNiFi Java
> - various flows I have are running as expected
> - confirmed the fix that blocked the previous RC
>
> Thanks Joe for taking care of the RM duties and thanks to everyone that
> contributed in this release.
>
> Pierre
>
> 2018-03-28 11:46 GMT+02:00 Koji Kawamura :
>
> > +1 (binding)
> >
> > - Confirmed hashes
> > - Built with include-atlas profile
> > - Confirmed various flows with 3 node secured cluster on Ubuntu
> > - Tested integration with Hadoop environment and NiFi Registry
> >
> > Koji
> >
> > On Wed, Mar 28, 2018 at 12:27 PM, Andrew Lim  >
> > wrote:
> > > +1 (non-binding)
> > >
> > > -Ran full clean install on OS X (10.11.6)
> > > -Tested integration with Secure NiFi Registry (1.5.0)
> > > -Tested fine grained restricted component policies.  Verified two
> issues
> > discovered while testing RC1 have been fixed in RC2 [1, 2]
> > > -Ran basic flows successfully
> > > -Reviewed documentation
> > >
> > > Drew
> > >
> > > [1] https://issues.apache.org/jira/browse/NIFI-5008
> > > [2] https://issues.apache.org/jira/browse/NIFI-5009
> > >
> > >
> > >> On Mar 26, 2018, at 11:34 PM, Joe Witt  wrote:
> > >>
> > >> Hello,
> > >>
> > >> I am pleased to be calling this vote for the source release of Apache
> > >> NiFi nifi-1.6.0.
> > >>
> > >> The source zip, including signatures, digests, etc. can be found at:
> > >> https://repository.apache.org/content/repositories/orgapachenifi-1123
> > >>
> > >> The Git tag is nifi-1.6.0-RC2
> > >> The Git commit ID is b5935ec81a7cbc048820781ac62cd96bbea5b232
> > >> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> > b5935ec81a7cbc048820781ac62cd96bbea5b232
> > >>
> > >> Checksums of nifi-1.6.0-source-release.zip:
> > >> SHA1: 009f1e2e3c17e38f21f27170b9c06228d11653c0
> > >> SHA256: 39941a5b25427e2b4cc5ba8206084ff92df58863f29ddd097d4ac1e85424
> > beb9
> > >> SHA512: 1773417a48665e3cda22180ea7f401bc8190ebddbf3f7bc29831e46e7ab0
> > a07694c6e478d252fa573209d4a3c8132a522a8507b6a8784669ab7364847a07e234
> > >>
> > >> Release artifacts are signed with the following key:
> > >> https://people.apache.org/keys/committer/joewitt.asc
> > >>
> > >> KEYS file available here:
> > >> https://dist.apache.org/repos/dist/release/nifi/KEYS
> > >>
> > >> 146 issues were closed/resolved for this release:
> > >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > projectId=12316020=12342422
> > >>
> > >> Release note highlights can be found here:
> > >> https://cwiki.apache.org/confluence/display/NIFI/
> > Release+Notes#ReleaseNotes-Version1.6.0
> > >>
> > >> The vote will be open for 72 hours.
> > >> Please download the release candidate and evaluate the necessary items
> > >> including checking hashes, signatures, build
> > >> from source, and test.  The please vote:
> > >>
> > >> [ ] +1 Release this package as nifi-1.6.0
> > >> [ ] +0 no opinion
> > >> [ ] -1 Do not release this package because...
> > >
> >
>

Re: [VOTE] Release Apache NiFi 1.6.0 (RC2)

[Pierre Villard]

+1 (binding)

- went through the release helper guide
- full clean install on OS X
- test unsecured and secured cluster setups and communications with
Registry and MiNiFi Java
- various flows I have are running as expected
- confirmed the fix that blocked the previous RC

Thanks Joe for taking care of the RM duties and thanks to everyone that
contributed in this release.

Pierre

2018-03-28 11:46 GMT+02:00 Koji Kawamura :

> +1 (binding)
>
> - Confirmed hashes
> - Built with include-atlas profile
> - Confirmed various flows with 3 node secured cluster on Ubuntu
> - Tested integration with Hadoop environment and NiFi Registry
>
> Koji
>
> On Wed, Mar 28, 2018 at 12:27 PM, Andrew Lim 
> wrote:
> > +1 (non-binding)
> >
> > -Ran full clean install on OS X (10.11.6)
> > -Tested integration with Secure NiFi Registry (1.5.0)
> > -Tested fine grained restricted component policies.  Verified two issues
> discovered while testing RC1 have been fixed in RC2 [1, 2]
> > -Ran basic flows successfully
> > -Reviewed documentation
> >
> > Drew
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-5008
> > [2] https://issues.apache.org/jira/browse/NIFI-5009
> >
> >
> >> On Mar 26, 2018, at 11:34 PM, Joe Witt  wrote:
> >>
> >> Hello,
> >>
> >> I am pleased to be calling this vote for the source release of Apache
> >> NiFi nifi-1.6.0.
> >>
> >> The source zip, including signatures, digests, etc. can be found at:
> >> https://repository.apache.org/content/repositories/orgapachenifi-1123
> >>
> >> The Git tag is nifi-1.6.0-RC2
> >> The Git commit ID is b5935ec81a7cbc048820781ac62cd96bbea5b232
> >> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> b5935ec81a7cbc048820781ac62cd96bbea5b232
> >>
> >> Checksums of nifi-1.6.0-source-release.zip:
> >> SHA1: 009f1e2e3c17e38f21f27170b9c06228d11653c0
> >> SHA256: 39941a5b25427e2b4cc5ba8206084ff92df58863f29ddd097d4ac1e85424
> beb9
> >> SHA512: 1773417a48665e3cda22180ea7f401bc8190ebddbf3f7bc29831e46e7ab0
> a07694c6e478d252fa573209d4a3c8132a522a8507b6a8784669ab7364847a07e234
> >>
> >> Release artifacts are signed with the following key:
> >> https://people.apache.org/keys/committer/joewitt.asc
> >>
> >> KEYS file available here:
> >> https://dist.apache.org/repos/dist/release/nifi/KEYS
> >>
> >> 146 issues were closed/resolved for this release:
> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12316020=12342422
> >>
> >> Release note highlights can be found here:
> >> https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-Version1.6.0
> >>
> >> The vote will be open for 72 hours.
> >> Please download the release candidate and evaluate the necessary items
> >> including checking hashes, signatures, build
> >> from source, and test.  The please vote:
> >>
> >> [ ] +1 Release this package as nifi-1.6.0
> >> [ ] +0 no opinion
> >> [ ] -1 Do not release this package because...
> >
>

Re: [VOTE] Release Apache NiFi 1.6.0 (RC2)

[Koji Kawamura]

+1 (binding)

- Confirmed hashes
- Built with include-atlas profile
- Confirmed various flows with 3 node secured cluster on Ubuntu
- Tested integration with Hadoop environment and NiFi Registry

Koji

On Wed, Mar 28, 2018 at 12:27 PM, Andrew Lim  wrote:
> +1 (non-binding)
>
> -Ran full clean install on OS X (10.11.6)
> -Tested integration with Secure NiFi Registry (1.5.0)
> -Tested fine grained restricted component policies.  Verified two issues 
> discovered while testing RC1 have been fixed in RC2 [1, 2]
> -Ran basic flows successfully
> -Reviewed documentation
>
> Drew
>
> [1] https://issues.apache.org/jira/browse/NIFI-5008
> [2] https://issues.apache.org/jira/browse/NIFI-5009
>
>
>> On Mar 26, 2018, at 11:34 PM, Joe Witt  wrote:
>>
>> Hello,
>>
>> I am pleased to be calling this vote for the source release of Apache
>> NiFi nifi-1.6.0.
>>
>> The source zip, including signatures, digests, etc. can be found at:
>> https://repository.apache.org/content/repositories/orgapachenifi-1123
>>
>> The Git tag is nifi-1.6.0-RC2
>> The Git commit ID is b5935ec81a7cbc048820781ac62cd96bbea5b232
>> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=b5935ec81a7cbc048820781ac62cd96bbea5b232
>>
>> Checksums of nifi-1.6.0-source-release.zip:
>> SHA1: 009f1e2e3c17e38f21f27170b9c06228d11653c0
>> SHA256: 39941a5b25427e2b4cc5ba8206084ff92df58863f29ddd097d4ac1e85424beb9
>> SHA512: 
>> 1773417a48665e3cda22180ea7f401bc8190ebddbf3f7bc29831e46e7ab0a07694c6e478d252fa573209d4a3c8132a522a8507b6a8784669ab7364847a07e234
>>
>> Release artifacts are signed with the following key:
>> https://people.apache.org/keys/committer/joewitt.asc
>>
>> KEYS file available here:
>> https://dist.apache.org/repos/dist/release/nifi/KEYS
>>
>> 146 issues were closed/resolved for this release:
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12342422
>>
>> Release note highlights can be found here:
>> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.6.0
>>
>> The vote will be open for 72 hours.
>> Please download the release candidate and evaluate the necessary items
>> including checking hashes, signatures, build
>> from source, and test.  The please vote:
>>
>> [ ] +1 Release this package as nifi-1.6.0
>> [ ] +0 no opinion
>> [ ] -1 Do not release this package because...
>

Re: Rat errors building with mvn clean install -Pcontrib-check

[Pierre Villard]

A good way to clean the offending files is usually to do:
git clean -dn (to just list the files)
git clean -df (to actually delete the files)
Be sure to check if you're not going to remove not committed work...

Pierre

2018-03-27 18:07 GMT+02:00 Otto Fowler :

> Ok, that would certainly fit.
> Thanks!
>
>
>
> On March 27, 2018 at 11:30:09, Joe Percivall (jperciv...@apache.org)
> wrote:
>
> Hey Otto,
>
> I've run into this before and it's typically due to switching between
> branches where modules have been added/removed. If you look at the files
> it's complaining about they're all in a target folder. Those are typically
> ignored when the maven module is included (i.e. the nifi-hadoop-utils
> module). When you switch off of a branch that had that, the ignored files
> stick around (i.e. the target folder).
>
> Long story short, you didn't do anything wrong. Just a by-product of doing
> dev work and you just need to remove the offending files.
>
> Joe
>
> On Tue, Mar 27, 2018 at 11:02 AM, Otto Fowler 
> wrote:
>
> > I have a branch on my work where I have been doing processor work.
> > I ran mvn clean install -Pcontrib-check
> > and now I’m getting rat errors for nifi-commons, which I didn’t change at
> > all. As anyone see this?
> >
> > *
> > Summary
> > ---
> > Generated at: 2018-03-27T10:56:27-04:00
> >
> > Notes: 18
> > Binaries: 44
> > Archives: 2
> > Standards: 18
> >
> > Apache Licensed: 1
> > Generated Documents: 0
> >
> > JavaDocs are generated, thus a license header is optional.
> > Generated files do not require license headers.
> >
> > 15 Unknown Licenses
> >
> > *
> >
> > Files with unapproved licenses:
> >
> >
> >
> /Users/ottobackwards/src/apache/forks/nifi/nifi-commons/nifi-hadoop-utils/
> > target/test-classes/krb5.conf
> >
> >
> /Users/ottobackwards/src/apache/forks/nifi/nifi-commons/nifi-hadoop-utils/
> > target/maven-status/maven-compiler-plugin/testCompile/
> > groovy-tests/inputFiles.lst
> >
> >
> /Users/ottobackwards/src/apache/forks/nifi/nifi-commons/nifi-hadoop-utils/
> > target/maven-status/maven-compiler-plugin/testCompile/
> > groovy-tests/createdFiles.lst
> >
> >
> /Users/ottobackwards/src/apache/forks/nifi/nifi-commons/nifi-hadoop-utils/
> >
> target/maven-status/maven-compiler-plugin/testCompile/default-testCompile/
> > inputFiles.lst
> >
> >
> /Users/ottobackwards/src/apache/forks/nifi/nifi-commons/nifi-hadoop-utils/
> >
> target/maven-status/maven-compiler-plugin/testCompile/default-testCompile/
> > createdFiles.lst
> >
> >
> /Users/ottobackwards/src/apache/forks/nifi/nifi-commons/nifi-hadoop-utils/
> > target/maven-status/maven-compiler-plugin/compile/
> > default-compile/inputFiles.lst
> >
> >
> /Users/ottobackwards/src/apache/forks/nifi/nifi-commons/nifi-hadoop-utils/
> > target/maven-status/maven-compiler-plugin/compile/
> > default-compile/createdFiles.lst
> >
> >
> /Users/ottobackwards/src/apache/forks/nifi/nifi-commons/nifi-hadoop-utils/
> > target/surefire-reports/org.apache.nifi.hadoop.
> TestKerberosProperties.txt
> >
> >
> /Users/ottobackwards/src/apache/forks/nifi/nifi-commons/nifi-hadoop-utils/
> > target/surefire-reports/TEST-org.apache.nifi.hadoop.
> > TestKerberosProperties.xml
> >
> >
> /Users/ottobackwards/src/apache/forks/nifi/nifi-commons/nifi-hadoop-utils/
> > target/.plxarc
> >
> > /Users/ottobackwards/src/apache/forks/nifi/nifi-commons/nifi-processor-
> > utilities/target/maven-status/maven-compiler-plugin/
> > testCompile/groovy-tests/inputFiles.lst
> >
> > /Users/ottobackwards/src/apache/forks/nifi/nifi-commons/nifi-processor-
> > utilities/target/maven-status/maven-compiler-plugin/testCompile/default-
> > testCompile/inputFiles.lst
> >
> > /Users/ottobackwards/src/apache/forks/nifi/nifi-commons/nifi-processor-
> > utilities/target/maven-status/maven-compiler-plugin/compile/
> > default-compile/inputFiles.lst
> >
> > /Users/ottobackwards/src/apache/forks/nifi/nifi-commons/nifi-processor-
> > utilities/target/maven-status/maven-compiler-plugin/compile/
> > default-compile/createdFiles.lst
> >
> > /Users/ottobackwards/src/apache/forks/nifi/nifi-commons/nifi-processor-
> > utilities/target/.plxarc
> >
> > *
> >
>
>
>
> --
> *Joe Percivall*
> linkedin.com/in/Percivall
> e: jperciv...@apache.com
>