Re: [VOTE] Release Apache NiFi 1.6.0

2018-03-23 Thread Sivaprasanna 
Yeah. Tested and found the bug when using the Schema Registry service. I
have made the changes and raised PR #2581


I have tested that both the scenario works as expected:

   1. Original reason why the ticket was raised i.e. to reload new
   resources found
   2. The schema registry bug

@Bryan and others,

Please review and check it.

Thanks,

Sivaprasanna

On Sat, Mar 24, 2018 at 3:23 AM, Bryan Bende  wrote:

> After voting I happened to be using the RC to test something else and
> came across a bug that I think warrants changing my vote to a -1.
>
> I created a simple two node cluster and made a standard convert record
> flow. When I ran the flow I got a schema not found exception, so I
> used the debugger which showed AvroSchemaRegistry had no schemas, even
> though there was one in the UI.
>
> I then used the debugger to make sure the onPropertyModified was
> getting when a schema was added, and it was which meant some after
> adding the schema but before running the flow, it was being removed.
>
> As far as I can tell, the issue is related to changes introduced in
> NIFI-4864... the intent here was for components with property
> descriptors that have "dynamically modifies classpath" to be able to
> smartly reload when they are started based on knowing if more
> classpath resources were added.
>
> The issue is that for components that don't have any property
> descriptors like this, they have a null fingerprint, and before
> starting it compares null to the fingerprint of empty string, and
> decides to reload [2].
>
> I think the fix should be fairly easy to just short-circuit at the
> beginning of that method and return immediately if
> additionalResourcesFingerprint is null, but will have to do some
> testing.
>
> [1] https://issues.apache.org/jira/browse/NIFI-4864
> [2] https://github.com/apache/nifi/blob/master/nifi-nar-
> bundles/nifi-framework-bundle/nifi-framework/nifi-framework-
> core-api/src/main/java/org/apache/nifi/controller/
> AbstractConfiguredComponent.java#L313-L314
>
>
> On Fri, Mar 23, 2018 at 4:20 PM, Matt Gilman 
> wrote:
> > +1 (binding) Release this package as nifi-1.6.0
> >
> > Executed the release helper and verified new granular restrictions with
> > regards to flow versioning.
> >
> > Thanks for RMing Joe!
> >
> > Matt
> >
> > On Fri, Mar 23, 2018 at 4:12 PM, Michael Moser 
> wrote:
> >
> >> +1 (binding)
> >>
> >> Ran through release helper to verify the release and run NiFi on Ubuntu
> >> 16.04.  It worked as expected with no new comments to add.
> >>
> >> -- Mike
> >>
> >>
> >> On Fri, Mar 23, 2018 at 4:02 PM, Scott Aslan 
> >> wrote:
> >>
> >> > +1 (binding)
> >> >
> >> > - Ran through release helper
> >> > - Setup secure NiFi and verified a test flow
> >> >
> >> > On Fri, Mar 23, 2018 at 3:29 PM, Bryan Bende 
> wrote:
> >> >
> >> > > +1 (binding)
> >> > >
> >> > > - Ran through release helper and everything checked out
> >> > > - Verified some test flows with the restricted components + keytab
> CS
> >> > >
> >> > >
> >> > >
> >> > > On Fri, Mar 23, 2018 at 2:42 PM, Mark Payne 
> >> > wrote:
> >> > > > +1 (binding)
> >> > > >
> >> > > > Was able to verify hashes, build with contrib-check, and start up
> >> > > application. Performed some basic functionality tests and all
> worked as
> >> > > expected.
> >> > > >
> >> > > > Thanks!
> >> > > > -Mark
> >> > > >
> >> > > >
> >> > > >> On Mar 23, 2018, at 6:02 AM, Takanobu Asanuma <
> >> tasan...@yahoo-corp.jp
> >> > >
> >> > > wrote:
> >> > > >>
> >> > > >> Thanks for all your efforts, Joe.
> >> > > >>
> >> > > >> I have one question. The version of the generated package is
> >> > > nifi-1.7.0-SNAPSHOT. Is this correct at this stage? If it's ok,
> >> > > +1(non-binding).
> >> > > >>
> >> > > >> - Succeeded "mvn -T 2.0C clean install -DskipTests -Prpm"
> >> > > >> - Started secure cluster mode
> >> > > >> - Verified sample dataflows work fine
> >> > > >> - Verified the whitelist feature(NIFI-4761) works fine
> >> > > >>
> >> > > >> -Takanobu Asanuma
> >> > > >>
> >> > > >> -Original Message-
> >> > > >> From: Joe Witt [mailto:joew...@apache.org]
> >> > > >> Sent: Friday, March 23, 2018 1:12 AM
> >> > > >> To: dev@nifi.apache.org
> >> > > >> Subject: [VOTE] Release Apache NiFi 1.6.0
> >> > > >>
> >> > > >> Hello,
> >> > > >>
> >> > > >> I am pleased to be calling this vote for the source release of
> >> Apache
> >> > > NiFi nifi-1.6.0.
> >> > > >>
> >> > > >> The source zip, including signatures, digests, etc. can be found
> at:
> >> > > >> https://repository.apache.org/content/repositories/
> >> orgapachenifi-1122
> >> > > >>
> >> > > >> The Git tag is nifi-1.6.0-RC1
> >> > > >> The Git commit ID is 49a71f4740c9fac38958961f78dd3cde874b0e45
> >> > > >> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> >> > >

Re: [VOTE] Release Apache NiFi 1.6.0

2018-03-23 Thread Pierre Villard 
-1 (binding)

I confirm the issue mentioned by Bryan. That's actually what Matt and I
experienced when trying the PR about the S2S Metrics Reporting task [1]. I
thought it was due to my change but it appears it's not the case.

[1] https://github.com/apache/nifi/pull/2575

2018-03-23 22:53 GMT+01:00 Bryan Bende :

> After voting I happened to be using the RC to test something else and
> came across a bug that I think warrants changing my vote to a -1.
>
> I created a simple two node cluster and made a standard convert record
> flow. When I ran the flow I got a schema not found exception, so I
> used the debugger which showed AvroSchemaRegistry had no schemas, even
> though there was one in the UI.
>
> I then used the debugger to make sure the onPropertyModified was
> getting when a schema was added, and it was which meant some after
> adding the schema but before running the flow, it was being removed.
>
> As far as I can tell, the issue is related to changes introduced in
> NIFI-4864... the intent here was for components with property
> descriptors that have "dynamically modifies classpath" to be able to
> smartly reload when they are started based on knowing if more
> classpath resources were added.
>
> The issue is that for components that don't have any property
> descriptors like this, they have a null fingerprint, and before
> starting it compares null to the fingerprint of empty string, and
> decides to reload [2].
>
> I think the fix should be fairly easy to just short-circuit at the
> beginning of that method and return immediately if
> additionalResourcesFingerprint is null, but will have to do some
> testing.
>
> [1] https://issues.apache.org/jira/browse/NIFI-4864
> [2] https://github.com/apache/nifi/blob/master/nifi-nar-
> bundles/nifi-framework-bundle/nifi-framework/nifi-framework-
> core-api/src/main/java/org/apache/nifi/controller/
> AbstractConfiguredComponent.java#L313-L314
>
>
> On Fri, Mar 23, 2018 at 4:20 PM, Matt Gilman 
> wrote:
> > +1 (binding) Release this package as nifi-1.6.0
> >
> > Executed the release helper and verified new granular restrictions with
> > regards to flow versioning.
> >
> > Thanks for RMing Joe!
> >
> > Matt
> >
> > On Fri, Mar 23, 2018 at 4:12 PM, Michael Moser 
> wrote:
> >
> >> +1 (binding)
> >>
> >> Ran through release helper to verify the release and run NiFi on Ubuntu
> >> 16.04.  It worked as expected with no new comments to add.
> >>
> >> -- Mike
> >>
> >>
> >> On Fri, Mar 23, 2018 at 4:02 PM, Scott Aslan 
> >> wrote:
> >>
> >> > +1 (binding)
> >> >
> >> > - Ran through release helper
> >> > - Setup secure NiFi and verified a test flow
> >> >
> >> > On Fri, Mar 23, 2018 at 3:29 PM, Bryan Bende 
> wrote:
> >> >
> >> > > +1 (binding)
> >> > >
> >> > > - Ran through release helper and everything checked out
> >> > > - Verified some test flows with the restricted components + keytab
> CS
> >> > >
> >> > >
> >> > >
> >> > > On Fri, Mar 23, 2018 at 2:42 PM, Mark Payne 
> >> > wrote:
> >> > > > +1 (binding)
> >> > > >
> >> > > > Was able to verify hashes, build with contrib-check, and start up
> >> > > application. Performed some basic functionality tests and all
> worked as
> >> > > expected.
> >> > > >
> >> > > > Thanks!
> >> > > > -Mark
> >> > > >
> >> > > >
> >> > > >> On Mar 23, 2018, at 6:02 AM, Takanobu Asanuma <
> >> tasan...@yahoo-corp.jp
> >> > >
> >> > > wrote:
> >> > > >>
> >> > > >> Thanks for all your efforts, Joe.
> >> > > >>
> >> > > >> I have one question. The version of the generated package is
> >> > > nifi-1.7.0-SNAPSHOT. Is this correct at this stage? If it's ok,
> >> > > +1(non-binding).
> >> > > >>
> >> > > >> - Succeeded "mvn -T 2.0C clean install -DskipTests -Prpm"
> >> > > >> - Started secure cluster mode
> >> > > >> - Verified sample dataflows work fine
> >> > > >> - Verified the whitelist feature(NIFI-4761) works fine
> >> > > >>
> >> > > >> -Takanobu Asanuma
> >> > > >>
> >> > > >> -Original Message-
> >> > > >> From: Joe Witt [mailto:joew...@apache.org]
> >> > > >> Sent: Friday, March 23, 2018 1:12 AM
> >> > > >> To: dev@nifi.apache.org
> >> > > >> Subject: [VOTE] Release Apache NiFi 1.6.0
> >> > > >>
> >> > > >> Hello,
> >> > > >>
> >> > > >> I am pleased to be calling this vote for the source release of
> >> Apache
> >> > > NiFi nifi-1.6.0.
> >> > > >>
> >> > > >> The source zip, including signatures, digests, etc. can be found
> at:
> >> > > >> https://repository.apache.org/content/repositories/
> >> orgapachenifi-1122
> >> > > >>
> >> > > >> The Git tag is nifi-1.6.0-RC1
> >> > > >> The Git commit ID is 49a71f4740c9fac38958961f78dd3cde874b0e45
> >> > > >> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> >> > > 49a71f4740c9fac38958961f78dd3cde874b0e45
> >> > > >>
> >> > > >> Checksums of nifi-1.6.0-source-release.zip:
> >> > > >> SHA1: 4fb82f386a0aa83614cc01449e540527443811ce
>

Re: SOAP Service through InvokeHTTP

2018-03-23 Thread Code Novice 
I've only just subscribed to the dev@nifi.apache.org mailing list and today
was my first time even knowing that a thing like 'maling lists' existed. 
Regardless, I'm posting my question in hopes to seek out how you eventually
managed to get your InvokeHTTP Processor to work with SOAP as I have not yet
been able to do so.

I posted my original question on this matter on Stackoverflow  here

 
. I also referenced this post as it is the only place I am seeing somebody
use the InvokeHTTP with SOAP.

I'm using SOAPUI to verify that everything works and it does as I am getting
the appropriate XML response back. However I am unable to accomplish this in
NiFi. My guess is that I don't know what to call my dynamic properties. I
also don't fully understand what data I would add to the Value of my dynamic
properties.

 



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: [VOTE] Release Apache NiFi 1.6.0

2018-03-23 Thread Bryan Bende 
After voting I happened to be using the RC to test something else and
came across a bug that I think warrants changing my vote to a -1.

I created a simple two node cluster and made a standard convert record
flow. When I ran the flow I got a schema not found exception, so I
used the debugger which showed AvroSchemaRegistry had no schemas, even
though there was one in the UI.

I then used the debugger to make sure the onPropertyModified was
getting when a schema was added, and it was which meant some after
adding the schema but before running the flow, it was being removed.

As far as I can tell, the issue is related to changes introduced in
NIFI-4864... the intent here was for components with property
descriptors that have "dynamically modifies classpath" to be able to
smartly reload when they are started based on knowing if more
classpath resources were added.

The issue is that for components that don't have any property
descriptors like this, they have a null fingerprint, and before
starting it compares null to the fingerprint of empty string, and
decides to reload [2].

I think the fix should be fairly easy to just short-circuit at the
beginning of that method and return immediately if
additionalResourcesFingerprint is null, but will have to do some
testing.

[1] https://issues.apache.org/jira/browse/NIFI-4864
[2] 
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/AbstractConfiguredComponent.java#L313-L314


On Fri, Mar 23, 2018 at 4:20 PM, Matt Gilman  wrote:
> +1 (binding) Release this package as nifi-1.6.0
>
> Executed the release helper and verified new granular restrictions with
> regards to flow versioning.
>
> Thanks for RMing Joe!
>
> Matt
>
> On Fri, Mar 23, 2018 at 4:12 PM, Michael Moser  wrote:
>
>> +1 (binding)
>>
>> Ran through release helper to verify the release and run NiFi on Ubuntu
>> 16.04.  It worked as expected with no new comments to add.
>>
>> -- Mike
>>
>>
>> On Fri, Mar 23, 2018 at 4:02 PM, Scott Aslan 
>> wrote:
>>
>> > +1 (binding)
>> >
>> > - Ran through release helper
>> > - Setup secure NiFi and verified a test flow
>> >
>> > On Fri, Mar 23, 2018 at 3:29 PM, Bryan Bende  wrote:
>> >
>> > > +1 (binding)
>> > >
>> > > - Ran through release helper and everything checked out
>> > > - Verified some test flows with the restricted components + keytab CS
>> > >
>> > >
>> > >
>> > > On Fri, Mar 23, 2018 at 2:42 PM, Mark Payne 
>> > wrote:
>> > > > +1 (binding)
>> > > >
>> > > > Was able to verify hashes, build with contrib-check, and start up
>> > > application. Performed some basic functionality tests and all worked as
>> > > expected.
>> > > >
>> > > > Thanks!
>> > > > -Mark
>> > > >
>> > > >
>> > > >> On Mar 23, 2018, at 6:02 AM, Takanobu Asanuma <
>> tasan...@yahoo-corp.jp
>> > >
>> > > wrote:
>> > > >>
>> > > >> Thanks for all your efforts, Joe.
>> > > >>
>> > > >> I have one question. The version of the generated package is
>> > > nifi-1.7.0-SNAPSHOT. Is this correct at this stage? If it's ok,
>> > > +1(non-binding).
>> > > >>
>> > > >> - Succeeded "mvn -T 2.0C clean install -DskipTests -Prpm"
>> > > >> - Started secure cluster mode
>> > > >> - Verified sample dataflows work fine
>> > > >> - Verified the whitelist feature(NIFI-4761) works fine
>> > > >>
>> > > >> -Takanobu Asanuma
>> > > >>
>> > > >> -Original Message-
>> > > >> From: Joe Witt [mailto:joew...@apache.org]
>> > > >> Sent: Friday, March 23, 2018 1:12 AM
>> > > >> To: dev@nifi.apache.org
>> > > >> Subject: [VOTE] Release Apache NiFi 1.6.0
>> > > >>
>> > > >> Hello,
>> > > >>
>> > > >> I am pleased to be calling this vote for the source release of
>> Apache
>> > > NiFi nifi-1.6.0.
>> > > >>
>> > > >> The source zip, including signatures, digests, etc. can be found at:
>> > > >> https://repository.apache.org/content/repositories/
>> orgapachenifi-1122
>> > > >>
>> > > >> The Git tag is nifi-1.6.0-RC1
>> > > >> The Git commit ID is 49a71f4740c9fac38958961f78dd3cde874b0e45
>> > > >> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
>> > > 49a71f4740c9fac38958961f78dd3cde874b0e45
>> > > >>
>> > > >> Checksums of nifi-1.6.0-source-release.zip:
>> > > >> SHA1: 4fb82f386a0aa83614cc01449e540527443811ce
>> > > >> SHA256: 4df6638ec87a5bee12e7978abc6413
>> 7ee5da5fc8c399e34cf34ca1c3720a
>> > > c891
>> > > >> SHA512: 6fa536f9618c6c153c04df5db59913
>> eaf3dd54ae2389368129ac6237f755
>> > > 19e1eee7ba0ca70145a95de01517a1c3ea1f36975895d9d72bb04abb56b7934e013a
>> > > >>
>> > > >> Release artifacts are signed with the following key:
>> > > >> https://people.apache.org/keys/committer/joewitt.asc
>> > > >>
>> > > >> KEYS file available here:
>> > > >> https://dist.apache.org/repos/dist/release/nifi/KEYS
>> > > >>
>> > > >> 135 issues were closed/resolved for this release:
>> > > >>

Objection: LDAP Auth needs HTTPS

2018-03-23 Thread Qcho 
Hi,

I'm working on deploying a nifi cluster with kubernetes.

The idea was to place a nifi-cluster behind an nginx-ingress.

We want to be able to access:

nifi.mydomain.com and allow load-balancing over:

node1.nifi.mydomain.com
node2.nifi.mydomain.com
... etc

The problem here is that we want to perform SSL termination in the
load-balancer at nifi.mydomain.com and then just talk plain http with each
node. But then it seems Nifi is not allowing this because it thinks talking
over http is always insecure and therefore no LDAP auth is supported.

Any though on this? Anyone has been able to do some kind of proxying??

Maybe we can add an enable-if-you-know-what-you-are-doing config for
allowing http auth?

Thanks in advance!

Re: [VOTE] Release Apache NiFi 1.6.0

2018-03-23 Thread Matt Gilman 
+1 (binding) Release this package as nifi-1.6.0

Executed the release helper and verified new granular restrictions with
regards to flow versioning.

Thanks for RMing Joe!

Matt

On Fri, Mar 23, 2018 at 4:12 PM, Michael Moser  wrote:

> +1 (binding)
>
> Ran through release helper to verify the release and run NiFi on Ubuntu
> 16.04.  It worked as expected with no new comments to add.
>
> -- Mike
>
>
> On Fri, Mar 23, 2018 at 4:02 PM, Scott Aslan 
> wrote:
>
> > +1 (binding)
> >
> > - Ran through release helper
> > - Setup secure NiFi and verified a test flow
> >
> > On Fri, Mar 23, 2018 at 3:29 PM, Bryan Bende  wrote:
> >
> > > +1 (binding)
> > >
> > > - Ran through release helper and everything checked out
> > > - Verified some test flows with the restricted components + keytab CS
> > >
> > >
> > >
> > > On Fri, Mar 23, 2018 at 2:42 PM, Mark Payne 
> > wrote:
> > > > +1 (binding)
> > > >
> > > > Was able to verify hashes, build with contrib-check, and start up
> > > application. Performed some basic functionality tests and all worked as
> > > expected.
> > > >
> > > > Thanks!
> > > > -Mark
> > > >
> > > >
> > > >> On Mar 23, 2018, at 6:02 AM, Takanobu Asanuma <
> tasan...@yahoo-corp.jp
> > >
> > > wrote:
> > > >>
> > > >> Thanks for all your efforts, Joe.
> > > >>
> > > >> I have one question. The version of the generated package is
> > > nifi-1.7.0-SNAPSHOT. Is this correct at this stage? If it's ok,
> > > +1(non-binding).
> > > >>
> > > >> - Succeeded "mvn -T 2.0C clean install -DskipTests -Prpm"
> > > >> - Started secure cluster mode
> > > >> - Verified sample dataflows work fine
> > > >> - Verified the whitelist feature(NIFI-4761) works fine
> > > >>
> > > >> -Takanobu Asanuma
> > > >>
> > > >> -Original Message-
> > > >> From: Joe Witt [mailto:joew...@apache.org]
> > > >> Sent: Friday, March 23, 2018 1:12 AM
> > > >> To: dev@nifi.apache.org
> > > >> Subject: [VOTE] Release Apache NiFi 1.6.0
> > > >>
> > > >> Hello,
> > > >>
> > > >> I am pleased to be calling this vote for the source release of
> Apache
> > > NiFi nifi-1.6.0.
> > > >>
> > > >> The source zip, including signatures, digests, etc. can be found at:
> > > >> https://repository.apache.org/content/repositories/
> orgapachenifi-1122
> > > >>
> > > >> The Git tag is nifi-1.6.0-RC1
> > > >> The Git commit ID is 49a71f4740c9fac38958961f78dd3cde874b0e45
> > > >> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> > > 49a71f4740c9fac38958961f78dd3cde874b0e45
> > > >>
> > > >> Checksums of nifi-1.6.0-source-release.zip:
> > > >> SHA1: 4fb82f386a0aa83614cc01449e540527443811ce
> > > >> SHA256: 4df6638ec87a5bee12e7978abc6413
> 7ee5da5fc8c399e34cf34ca1c3720a
> > > c891
> > > >> SHA512: 6fa536f9618c6c153c04df5db59913
> eaf3dd54ae2389368129ac6237f755
> > > 19e1eee7ba0ca70145a95de01517a1c3ea1f36975895d9d72bb04abb56b7934e013a
> > > >>
> > > >> Release artifacts are signed with the following key:
> > > >> https://people.apache.org/keys/committer/joewitt.asc
> > > >>
> > > >> KEYS file available here:
> > > >> https://dist.apache.org/repos/dist/release/nifi/KEYS
> > > >>
> > > >> 135 issues were closed/resolved for this release:
> > > >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > > projectId=12316020=12342422
> > > >>
> > > >> Release note highlights can be found here:
> > > >> https://cwiki.apache.org/confluence/display/NIFI/
> > > Release+Notes#ReleaseNotes-Version1.6.0
> > > >>
> > > >> The vote will be open for 72 hours.
> > > >> Please download the release candidate and evaluate the necessary
> items
> > > including checking hashes, signatures, build from source, and test.
> The
> > > please vote:
> > > >>
> > > >> [ ] +1 Release this package as nifi-1.6.0 [ ] +0 no opinion [ ] -1
> Do
> > > not release this package because...
> > > >
> > >
> >
>

Re: NiFi Versioned Process Group Status Icons

2018-03-23 Thread Matt Gilman 
Peter,

The status icon for a specific Process Group is hidden until that group is
versioned. That icon is positioned next to the group name.

For the icons in the status bar and in the bottom of the Process Group, we
were ultimately just trying to remain consistent. These reflect the counts
of the encapsulated versioned Process Groups and does not include itself.
Like we have with the status icons for the encapsulated Processors and
Ports. Even if this Process Group is not configured to have any
encapsulated components, we still render the counts as zero.

Additionally, the presence of the icons helps drive awareness of the
feature.

Rob or Drew may have some additional insight to add. Hope this helps

Matt

On Fri, Mar 23, 2018 at 12:10 AM, Peter Wicks (pwicks) 
wrote:

> Why does NiFi show status icons for Versioned Process Group's on servers
> that are not configured to connect to a NiFi Registry?
>
> Thanks,
>   Peter
>

Re: [VOTE] Release Apache NiFi 1.6.0

2018-03-23 Thread Michael Moser 
+1 (binding)

Ran through release helper to verify the release and run NiFi on Ubuntu
16.04.  It worked as expected with no new comments to add.

-- Mike


On Fri, Mar 23, 2018 at 4:02 PM, Scott Aslan  wrote:

> +1 (binding)
>
> - Ran through release helper
> - Setup secure NiFi and verified a test flow
>
> On Fri, Mar 23, 2018 at 3:29 PM, Bryan Bende  wrote:
>
> > +1 (binding)
> >
> > - Ran through release helper and everything checked out
> > - Verified some test flows with the restricted components + keytab CS
> >
> >
> >
> > On Fri, Mar 23, 2018 at 2:42 PM, Mark Payne 
> wrote:
> > > +1 (binding)
> > >
> > > Was able to verify hashes, build with contrib-check, and start up
> > application. Performed some basic functionality tests and all worked as
> > expected.
> > >
> > > Thanks!
> > > -Mark
> > >
> > >
> > >> On Mar 23, 2018, at 6:02 AM, Takanobu Asanuma  >
> > wrote:
> > >>
> > >> Thanks for all your efforts, Joe.
> > >>
> > >> I have one question. The version of the generated package is
> > nifi-1.7.0-SNAPSHOT. Is this correct at this stage? If it's ok,
> > +1(non-binding).
> > >>
> > >> - Succeeded "mvn -T 2.0C clean install -DskipTests -Prpm"
> > >> - Started secure cluster mode
> > >> - Verified sample dataflows work fine
> > >> - Verified the whitelist feature(NIFI-4761) works fine
> > >>
> > >> -Takanobu Asanuma
> > >>
> > >> -Original Message-
> > >> From: Joe Witt [mailto:joew...@apache.org]
> > >> Sent: Friday, March 23, 2018 1:12 AM
> > >> To: dev@nifi.apache.org
> > >> Subject: [VOTE] Release Apache NiFi 1.6.0
> > >>
> > >> Hello,
> > >>
> > >> I am pleased to be calling this vote for the source release of Apache
> > NiFi nifi-1.6.0.
> > >>
> > >> The source zip, including signatures, digests, etc. can be found at:
> > >> https://repository.apache.org/content/repositories/orgapachenifi-1122
> > >>
> > >> The Git tag is nifi-1.6.0-RC1
> > >> The Git commit ID is 49a71f4740c9fac38958961f78dd3cde874b0e45
> > >> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> > 49a71f4740c9fac38958961f78dd3cde874b0e45
> > >>
> > >> Checksums of nifi-1.6.0-source-release.zip:
> > >> SHA1: 4fb82f386a0aa83614cc01449e540527443811ce
> > >> SHA256: 4df6638ec87a5bee12e7978abc64137ee5da5fc8c399e34cf34ca1c3720a
> > c891
> > >> SHA512: 6fa536f9618c6c153c04df5db59913eaf3dd54ae2389368129ac6237f755
> > 19e1eee7ba0ca70145a95de01517a1c3ea1f36975895d9d72bb04abb56b7934e013a
> > >>
> > >> Release artifacts are signed with the following key:
> > >> https://people.apache.org/keys/committer/joewitt.asc
> > >>
> > >> KEYS file available here:
> > >> https://dist.apache.org/repos/dist/release/nifi/KEYS
> > >>
> > >> 135 issues were closed/resolved for this release:
> > >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > projectId=12316020=12342422
> > >>
> > >> Release note highlights can be found here:
> > >> https://cwiki.apache.org/confluence/display/NIFI/
> > Release+Notes#ReleaseNotes-Version1.6.0
> > >>
> > >> The vote will be open for 72 hours.
> > >> Please download the release candidate and evaluate the necessary items
> > including checking hashes, signatures, build from source, and test.  The
> > please vote:
> > >>
> > >> [ ] +1 Release this package as nifi-1.6.0 [ ] +0 no opinion [ ] -1 Do
> > not release this package because...
> > >
> >
>

Re: [VOTE] Release Apache NiFi 1.6.0

2018-03-23 Thread Scott Aslan 
+1 (binding)

- Ran through release helper
- Setup secure NiFi and verified a test flow

On Fri, Mar 23, 2018 at 3:29 PM, Bryan Bende  wrote:

> +1 (binding)
>
> - Ran through release helper and everything checked out
> - Verified some test flows with the restricted components + keytab CS
>
>
>
> On Fri, Mar 23, 2018 at 2:42 PM, Mark Payne  wrote:
> > +1 (binding)
> >
> > Was able to verify hashes, build with contrib-check, and start up
> application. Performed some basic functionality tests and all worked as
> expected.
> >
> > Thanks!
> > -Mark
> >
> >
> >> On Mar 23, 2018, at 6:02 AM, Takanobu Asanuma 
> wrote:
> >>
> >> Thanks for all your efforts, Joe.
> >>
> >> I have one question. The version of the generated package is
> nifi-1.7.0-SNAPSHOT. Is this correct at this stage? If it's ok,
> +1(non-binding).
> >>
> >> - Succeeded "mvn -T 2.0C clean install -DskipTests -Prpm"
> >> - Started secure cluster mode
> >> - Verified sample dataflows work fine
> >> - Verified the whitelist feature(NIFI-4761) works fine
> >>
> >> -Takanobu Asanuma
> >>
> >> -Original Message-
> >> From: Joe Witt [mailto:joew...@apache.org]
> >> Sent: Friday, March 23, 2018 1:12 AM
> >> To: dev@nifi.apache.org
> >> Subject: [VOTE] Release Apache NiFi 1.6.0
> >>
> >> Hello,
> >>
> >> I am pleased to be calling this vote for the source release of Apache
> NiFi nifi-1.6.0.
> >>
> >> The source zip, including signatures, digests, etc. can be found at:
> >> https://repository.apache.org/content/repositories/orgapachenifi-1122
> >>
> >> The Git tag is nifi-1.6.0-RC1
> >> The Git commit ID is 49a71f4740c9fac38958961f78dd3cde874b0e45
> >> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> 49a71f4740c9fac38958961f78dd3cde874b0e45
> >>
> >> Checksums of nifi-1.6.0-source-release.zip:
> >> SHA1: 4fb82f386a0aa83614cc01449e540527443811ce
> >> SHA256: 4df6638ec87a5bee12e7978abc64137ee5da5fc8c399e34cf34ca1c3720a
> c891
> >> SHA512: 6fa536f9618c6c153c04df5db59913eaf3dd54ae2389368129ac6237f755
> 19e1eee7ba0ca70145a95de01517a1c3ea1f36975895d9d72bb04abb56b7934e013a
> >>
> >> Release artifacts are signed with the following key:
> >> https://people.apache.org/keys/committer/joewitt.asc
> >>
> >> KEYS file available here:
> >> https://dist.apache.org/repos/dist/release/nifi/KEYS
> >>
> >> 135 issues were closed/resolved for this release:
> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12316020=12342422
> >>
> >> Release note highlights can be found here:
> >> https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-Version1.6.0
> >>
> >> The vote will be open for 72 hours.
> >> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.  The
> please vote:
> >>
> >> [ ] +1 Release this package as nifi-1.6.0 [ ] +0 no opinion [ ] -1 Do
> not release this package because...
> >
>

Re: [VOTE] Release Apache NiFi 1.6.0

2018-03-23 Thread Bryan Bende 
+1 (binding)

- Ran through release helper and everything checked out
- Verified some test flows with the restricted components + keytab CS



On Fri, Mar 23, 2018 at 2:42 PM, Mark Payne  wrote:
> +1 (binding)
>
> Was able to verify hashes, build with contrib-check, and start up 
> application. Performed some basic functionality tests and all worked as 
> expected.
>
> Thanks!
> -Mark
>
>
>> On Mar 23, 2018, at 6:02 AM, Takanobu Asanuma  wrote:
>>
>> Thanks for all your efforts, Joe.
>>
>> I have one question. The version of the generated package is 
>> nifi-1.7.0-SNAPSHOT. Is this correct at this stage? If it's ok, 
>> +1(non-binding).
>>
>> - Succeeded "mvn -T 2.0C clean install -DskipTests -Prpm"
>> - Started secure cluster mode
>> - Verified sample dataflows work fine
>> - Verified the whitelist feature(NIFI-4761) works fine
>>
>> -Takanobu Asanuma
>>
>> -Original Message-
>> From: Joe Witt [mailto:joew...@apache.org]
>> Sent: Friday, March 23, 2018 1:12 AM
>> To: dev@nifi.apache.org
>> Subject: [VOTE] Release Apache NiFi 1.6.0
>>
>> Hello,
>>
>> I am pleased to be calling this vote for the source release of Apache NiFi 
>> nifi-1.6.0.
>>
>> The source zip, including signatures, digests, etc. can be found at:
>> https://repository.apache.org/content/repositories/orgapachenifi-1122
>>
>> The Git tag is nifi-1.6.0-RC1
>> The Git commit ID is 49a71f4740c9fac38958961f78dd3cde874b0e45
>> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=49a71f4740c9fac38958961f78dd3cde874b0e45
>>
>> Checksums of nifi-1.6.0-source-release.zip:
>> SHA1: 4fb82f386a0aa83614cc01449e540527443811ce
>> SHA256: 4df6638ec87a5bee12e7978abc64137ee5da5fc8c399e34cf34ca1c3720ac891
>> SHA512: 
>> 6fa536f9618c6c153c04df5db59913eaf3dd54ae2389368129ac6237f75519e1eee7ba0ca70145a95de01517a1c3ea1f36975895d9d72bb04abb56b7934e013a
>>
>> Release artifacts are signed with the following key:
>> https://people.apache.org/keys/committer/joewitt.asc
>>
>> KEYS file available here:
>> https://dist.apache.org/repos/dist/release/nifi/KEYS
>>
>> 135 issues were closed/resolved for this release:
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12342422
>>
>> Release note highlights can be found here:
>> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.6.0
>>
>> The vote will be open for 72 hours.
>> Please download the release candidate and evaluate the necessary items 
>> including checking hashes, signatures, build from source, and test.  The 
>> please vote:
>>
>> [ ] +1 Release this package as nifi-1.6.0 [ ] +0 no opinion [ ] -1 Do not 
>> release this package because...
>

Re: [VOTE] Release Apache NiFi 1.6.0

2018-03-23 Thread Mark Payne 
+1 (binding)

Was able to verify hashes, build with contrib-check, and start up application. 
Performed some basic functionality tests and all worked as expected.

Thanks!
-Mark


> On Mar 23, 2018, at 6:02 AM, Takanobu Asanuma  wrote:
> 
> Thanks for all your efforts, Joe. 
> 
> I have one question. The version of the generated package is 
> nifi-1.7.0-SNAPSHOT. Is this correct at this stage? If it's ok, 
> +1(non-binding).
> 
> - Succeeded "mvn -T 2.0C clean install -DskipTests -Prpm"
> - Started secure cluster mode
> - Verified sample dataflows work fine
> - Verified the whitelist feature(NIFI-4761) works fine
> 
> -Takanobu Asanuma
> 
> -Original Message-
> From: Joe Witt [mailto:joew...@apache.org] 
> Sent: Friday, March 23, 2018 1:12 AM
> To: dev@nifi.apache.org
> Subject: [VOTE] Release Apache NiFi 1.6.0
> 
> Hello,
> 
> I am pleased to be calling this vote for the source release of Apache NiFi 
> nifi-1.6.0.
> 
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1122
> 
> The Git tag is nifi-1.6.0-RC1
> The Git commit ID is 49a71f4740c9fac38958961f78dd3cde874b0e45
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=49a71f4740c9fac38958961f78dd3cde874b0e45
> 
> Checksums of nifi-1.6.0-source-release.zip:
> SHA1: 4fb82f386a0aa83614cc01449e540527443811ce
> SHA256: 4df6638ec87a5bee12e7978abc64137ee5da5fc8c399e34cf34ca1c3720ac891
> SHA512: 
> 6fa536f9618c6c153c04df5db59913eaf3dd54ae2389368129ac6237f75519e1eee7ba0ca70145a95de01517a1c3ea1f36975895d9d72bb04abb56b7934e013a
> 
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
> 
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
> 
> 135 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12342422
> 
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.6.0
> 
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items 
> including checking hashes, signatures, build from source, and test.  The 
> please vote:
> 
> [ ] +1 Release this package as nifi-1.6.0 [ ] +0 no opinion [ ] -1 Do not 
> release this package because...

Master failing in travis

2018-03-23 Thread Otto Fowler 
Anyone have an idea why Master is failing in travis?

Re: FlattenJson

2018-03-23 Thread Bryan Bende 
Most of the ideas discussed here are assuming there is one record per
flow file, which for any serious amount of data is not what you want
to do.

It might be better to have a new ExtractJsonToAttributes processor
that enforces limitations like a single json doc per flow file and all
flat fields, so if you don't split and flatten before hand then it
routes to failure.


On Fri, Mar 23, 2018 at 11:07 AM, Jorge Machado  wrote:
> So I’m pretty lost now, all the suggestions from Matt will not solve my 
> problem that I need to have all contents of a flow file as attritube key 
> -paired…
>
> A good place to have it would be on ConvertAvroToJSON so that it has a option 
> to say if it goes to attribute or to FlowFile, defaulting to Flowfile.
>
> Would be the Changed accepted  ? I would create a PR for it.
>
>
> Jorge Machado
>
>
>
>
>
>> On 20 Mar 2018, at 22:35, Otto Fowler  wrote:
>>
>> We could start with routeOnJsonPath and do the record path as the need
>> arises?
>>
>>
>> On March 20, 2018 at 16:06:34, Matt Burgess (mattyb...@apache.org) wrote:
>>
>> Rather than restricting it to JSONPath, perhaps we should have a
>> RouteOnRecordPath or RouteRecord using the RecordPath API? Even better
>> would be the ability to use RecordPath functions in QueryRecord, but
>> that involves digging into Calcite as well. I realize JSONPath might
>> have more capabilities than RecordPath at the moment, but it seems a
>> shame to force the user to convert to JSON to use a "RouteOnJSONPath"
>> processor, the record-aware processors are meant to replace that kind
>> of format-specific functionality.
>>
>> Regards,
>> Matt
>>
>> On Tue, Mar 20, 2018 at 12:19 PM, Sivaprasanna
>>  wrote:
>>> Like the idea that Otto suggested. RoutOnJSONPath makes more sense since
>>> making the flattened JSON write to attributes is restricted to that
>>> processor alone.
>>>
>>> On Tue, Mar 20, 2018 at 8:37 PM, Otto Fowler 
>>> wrote:
>>>
 Why not create a new processor that does routeOnJSONPath and works on
>> the
 flow file?


 On March 20, 2018 at 10:39:37, Jorge Machado (jom...@me.com) wrote:

 So that is what we actually are doing EvaluateJsonPath the problem with
 that is, that is hard to build something generic if we need to specify
>> each
 property by his name, that’s why this idea.

 Should I make a PR for this or is this to business specific ?


 Jorge Machado

> On 20 Mar 2018, at 15:30, Bryan Bende  wrote:
>
> Ok so I guess it depends whether you end up needing all 30 fields as
> attributes to achieve the logic in your flow, or if you only need a
> couple.
>
> If you only need a couple you could probably use EvaluateJsonPath
> after FlattenJson to extract just the couple of fields you need into
> attributes.
>
> If you need them all then I guess it makes sense to want the option to
> flatten into attributes.
>
> On Tue, Mar 20, 2018 at 10:14 AM, Jorge Machado  wrote:
>> From there on we use a lot of routeOnAttritutes and use that values
>> on
 sql queries to other tables like select * from someTable where
 id=${myExtractedAttribute}
>> To be honest I tryed JoltTransformJSON but I could not get it working
>> :)
>>
>> Jorge Machado
>>
>>
>>
>>
>>
>>> On 20 Mar 2018, at 15:12, Matt Burgess  wrote:
>>>
>>> I think Bryan is asking about what happens AFTER this part of the
>>> flow. For example, if you are doing routing you can use QueryRecord
>>> (and you won't need the SplitJson), if you are doing transformations
>>> you can use JoltTransformJSON (often without SplitJson as well),
>> etc.
>>>
>>> Regards,
>>> Matt
>>>
>>> On Tue, Mar 20, 2018 at 10:08 AM, Jorge Machado 
>> wrote:
 Hi Bryan,

 thanks for the help.
 Our Flow: ExecuteSql -> convertToJSON -> SplitJson -> ExecuteScript
 with attachedcode 1.

 We are now writting a custom processor that does this which is a
>> copy
 of FlattenJson but instead of putting the result into a flowfile we put
>> it
 into the attributes.
 That’s why I asked if it makes sense to contribute this back



 Attached code 1:

 import org.apache.commons.io.IOUtils
 import java.nio.charset.*
 def flowFile = session.get();
 if (flowFile == null) {
 return;
 }
 def slurper = new groovy.json.JsonSlurper()
 def attrs = [:] as Map
 session.read(flowFile,
 { inputStream ->
 def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
 def obj = slurper.parseText(text)
 obj.each {k,v ->
 if(v!=null && v.toString()!=""){

Re: FlattenJson

2018-03-23 Thread Jorge Machado 
So I’m pretty lost now, all the suggestions from Matt will not solve my problem 
that I need to have all contents of a flow file as attritube key -paired… 

A good place to have it would be on ConvertAvroToJSON so that it has a option 
to say if it goes to attribute or to FlowFile, defaulting to Flowfile.

Would be the Changed accepted  ? I would create a PR for it. 


Jorge Machado





> On 20 Mar 2018, at 22:35, Otto Fowler  wrote:
> 
> We could start with routeOnJsonPath and do the record path as the need
> arises?
> 
> 
> On March 20, 2018 at 16:06:34, Matt Burgess (mattyb...@apache.org) wrote:
> 
> Rather than restricting it to JSONPath, perhaps we should have a
> RouteOnRecordPath or RouteRecord using the RecordPath API? Even better
> would be the ability to use RecordPath functions in QueryRecord, but
> that involves digging into Calcite as well. I realize JSONPath might
> have more capabilities than RecordPath at the moment, but it seems a
> shame to force the user to convert to JSON to use a "RouteOnJSONPath"
> processor, the record-aware processors are meant to replace that kind
> of format-specific functionality.
> 
> Regards,
> Matt
> 
> On Tue, Mar 20, 2018 at 12:19 PM, Sivaprasanna
>  wrote:
>> Like the idea that Otto suggested. RoutOnJSONPath makes more sense since
>> making the flattened JSON write to attributes is restricted to that
>> processor alone.
>> 
>> On Tue, Mar 20, 2018 at 8:37 PM, Otto Fowler 
>> wrote:
>> 
>>> Why not create a new processor that does routeOnJSONPath and works on
> the
>>> flow file?
>>> 
>>> 
>>> On March 20, 2018 at 10:39:37, Jorge Machado (jom...@me.com) wrote:
>>> 
>>> So that is what we actually are doing EvaluateJsonPath the problem with
>>> that is, that is hard to build something generic if we need to specify
> each
>>> property by his name, that’s why this idea.
>>> 
>>> Should I make a PR for this or is this to business specific ?
>>> 
>>> 
>>> Jorge Machado
>>> 
 On 20 Mar 2018, at 15:30, Bryan Bende  wrote:
 
 Ok so I guess it depends whether you end up needing all 30 fields as
 attributes to achieve the logic in your flow, or if you only need a
 couple.
 
 If you only need a couple you could probably use EvaluateJsonPath
 after FlattenJson to extract just the couple of fields you need into
 attributes.
 
 If you need them all then I guess it makes sense to want the option to
 flatten into attributes.
 
 On Tue, Mar 20, 2018 at 10:14 AM, Jorge Machado  wrote:
> From there on we use a lot of routeOnAttritutes and use that values
> on
>>> sql queries to other tables like select * from someTable where
>>> id=${myExtractedAttribute}
> To be honest I tryed JoltTransformJSON but I could not get it working
> :)
> 
> Jorge Machado
> 
> 
> 
> 
> 
>> On 20 Mar 2018, at 15:12, Matt Burgess  wrote:
>> 
>> I think Bryan is asking about what happens AFTER this part of the
>> flow. For example, if you are doing routing you can use QueryRecord
>> (and you won't need the SplitJson), if you are doing transformations
>> you can use JoltTransformJSON (often without SplitJson as well),
> etc.
>> 
>> Regards,
>> Matt
>> 
>> On Tue, Mar 20, 2018 at 10:08 AM, Jorge Machado 
> wrote:
>>> Hi Bryan,
>>> 
>>> thanks for the help.
>>> Our Flow: ExecuteSql -> convertToJSON -> SplitJson -> ExecuteScript
>>> with attachedcode 1.
>>> 
>>> We are now writting a custom processor that does this which is a
> copy
>>> of FlattenJson but instead of putting the result into a flowfile we put
> it
>>> into the attributes.
>>> That’s why I asked if it makes sense to contribute this back
>>> 
>>> 
>>> 
>>> Attached code 1:
>>> 
>>> import org.apache.commons.io.IOUtils
>>> import java.nio.charset.*
>>> def flowFile = session.get();
>>> if (flowFile == null) {
>>> return;
>>> }
>>> def slurper = new groovy.json.JsonSlurper()
>>> def attrs = [:] as Map
>>> session.read(flowFile,
>>> { inputStream ->
>>> def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
>>> def obj = slurper.parseText(text)
>>> obj.each {k,v ->
>>> if(v!=null && v.toString()!=""){
>>> attrs[k] = v.toString()
>>> }
>>> }
>>> } as InputStreamCallback)
>>> flowFile = session.putAllAttributes(flowFile, attrs)
>>> session.transfer(flowFile, REL_SUCCESS)
>>> 
>>> some code removed
>>> 
>>> 
>>> Jorge Machado
>>> 
>>> 
>>> 
>>> 
>>> 
 On 20 Mar 2018, at 15:03, Bryan Bende  wrote:
 
 Ok it is still not clear what the reason for needing it in
> attributes
 is though... Is there another processor you are using after

RE: [VOTE] Release Apache NiFi 1.6.0

2018-03-23 Thread Takanobu Asanuma 
Thanks for all your efforts, Joe. 

I have one question. The version of the generated package is 
nifi-1.7.0-SNAPSHOT. Is this correct at this stage? If it's ok, +1(non-binding).

 - Succeeded "mvn -T 2.0C clean install -DskipTests -Prpm"
 - Started secure cluster mode
 - Verified sample dataflows work fine
 - Verified the whitelist feature(NIFI-4761) works fine

-Takanobu Asanuma

-Original Message-
From: Joe Witt [mailto:joew...@apache.org] 
Sent: Friday, March 23, 2018 1:12 AM
To: dev@nifi.apache.org
Subject: [VOTE] Release Apache NiFi 1.6.0

Hello,

I am pleased to be calling this vote for the source release of Apache NiFi 
nifi-1.6.0.

The source zip, including signatures, digests, etc. can be found at:
https://repository.apache.org/content/repositories/orgapachenifi-1122

The Git tag is nifi-1.6.0-RC1
The Git commit ID is 49a71f4740c9fac38958961f78dd3cde874b0e45
https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=49a71f4740c9fac38958961f78dd3cde874b0e45

Checksums of nifi-1.6.0-source-release.zip:
SHA1: 4fb82f386a0aa83614cc01449e540527443811ce
SHA256: 4df6638ec87a5bee12e7978abc64137ee5da5fc8c399e34cf34ca1c3720ac891
SHA512: 
6fa536f9618c6c153c04df5db59913eaf3dd54ae2389368129ac6237f75519e1eee7ba0ca70145a95de01517a1c3ea1f36975895d9d72bb04abb56b7934e013a

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/joewitt.asc

KEYS file available here:
https://dist.apache.org/repos/dist/release/nifi/KEYS

135 issues were closed/resolved for this release:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12342422

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.6.0

The vote will be open for 72 hours.
Please download the release candidate and evaluate the necessary items 
including checking hashes, signatures, build from source, and test.  The please 
vote:

[ ] +1 Release this package as nifi-1.6.0 [ ] +0 no opinion [ ] -1 Do not 
release this package because...