RE: [EXT] Re: Automated Testing of Flows

[Peter Wicks (pwicks)]

I don't even know if I need Jenkins to setup a fresh instance, that could be 
nice, but I could use my TEST instance and just upgrade it by hand anytime I 
want to test a new version. I could create a dedicated Process Group that my 
test scripts would use for deploying the templates.

I've worked with the REST API quite a bit, so I'd probably use that to 
configure the data sources, etc... Have the test runner be Java or Python.

From a completely different approach, I saw a recent PR to allow for NiFi to be 
started up from within another process. This could be an interesting approach 
if the API's are easily accessible. https://github.com/apache/nifi/pull/2251




-Original Message-
From: Scott Aslan [mailto:scottyas...@gmail.com] 
Sent: Friday, November 17, 2017 12:39
To: dev@nifi.apache.org
Subject: [EXT] Re: Automated Testing of Flows

Just brain storming here

You could use travis or Jenkins to setup a job where you use mvn and npm to 
build and start NiFi. Assuming the test template was available from within the 
environment maybe you could use the protractor e2e testing suite to script the 
clicks necessary to load the template and run the flow...

On Wed, Nov 15, 2017 at 8:07 PM, Peter Wicks (pwicks) 
wrote:

> I would like to find a way to automatically run integration/upgrade 
> tests on flows, rather than just one or two processors like the mock 
> framework supports. Preferably something Template driven, where I can 
> provide a template xml file, and an accompanying config file to setup 
> the test, then run the flow and examine the results.
>
> Has anyone worked on something like this?
>

Re: [DISCUSS] Apache NiFi MiNiFi C++ 0.3.0

[Aldrin Piri]

Marc,

Thanks for the update and rounding up these last few issues.  The timeline
sounds good to me and appreciate you taking on the RM duties.  I have some
notes and such that should go for updating/crafting the release guide for
C++ land.  Things have changed a fair bit since our 0.2.0 release, but I'll
look to get those on a wiki as a starting point and we can iterate from
there.

--aldrin

On Thu, Nov 16, 2017 at 9:44 PM, Marc  wrote:

> Everyone,
>   It looks as though the issues mentioned yesterday have been solved for
> 0.3.0. I apologize for being the long pole in addressing these issues in a
> release that has seen some great work by a great group of contributors.
>
>   A new ticket was created in response to an issue that helped expose
> MINIFICPP-293 [0]. A fix for that ticket, MINIFICPP-308 [2], has been
> submitted and is ready for review. I will try and go through the PR later
> this evening or early tomorrow morning before a flight and merge it. That
> will close out the release, unless anyone objects.
>
>   I'm unavailable the next few days ( and will be in an isolated area
> without internet) so the plan is to to tag and release Monday afternoon or
> Tuesday morning, unless the release is needed sooner.
>
>  Thanks,
>  Marc
>   [0]  https://issues.apache.org/jira/browse/MINIFICPP-293
>   [1]  https://issues.apache.org/jira/browse/MINIFICPP-309
>   [2] https://issues.apache.org/jira/browse/MINIFICPP-308
>
> On Wed, Nov 15, 2017 at 8:33 AM, Marc  wrote:
>
> > Everyone,
> > I've been getting through some apache LDAP issues, hence the delay;
> > however, we do have three remaining issues --
> > https://issues.apache.org/jira/projects/MINIFICPP/versions/12341640.
> >
> >One is still under development. We can move it, but that would mean we
> > can't compile in CENTOS 6. At this time my goal is to resolve the
> > blockers/criticals since they would alienate whole groups of users. I've
> > moved everything else from 0.3.0.
> >
> > On Thu, Nov 2, 2017 at 7:39 PM, Marc  wrote:
> >
> >> Aldrin,
> >>   +1 to the idea of making a release.
> >>
> >>   Happy to try my hand at RM duties.
> >>
> >>   Can't promise I'll do well at it, but I can promise that when I screw
> >> up, we'll all have a good laugh.
> >>
> >>
> >>
> >> On Wed, Nov 1, 2017 at 3:04 PM, Aldrin Piri 
> wrote:
> >>
> >>> Hey folks,
> >>>
> >>> We've had a fair amount of work and enhancements to the framework and
> >>> extensions and seemed like a logical place to offer up a new release.
> >>> Both
> >>> Execute Script and HTTP Site to Site seem to be nearing their merge
> into
> >>> master.  With a quick look through other open JIRAs it appears as some
> of
> >>> the newer tickets starting progress might be extended efforts.
> >>>
> >>> Let me know if anyone has any thoughts or comments.  Otherwise, we can
> >>> try
> >>> to generate an RC and vote in the next week.  I am happy to act as RM
> if
> >>> no
> >>> one else is interested in doing so (likely have some docs I need to
> >>> update
> >>> to capture the process a bit better).
> >>>
> >>>
> >>> --Aldrin
> >>>
> >>
> >>
> >
>

Re: build failure

[Aldrin Piri]

Hi Mark,

Not sure of the exact cause, but I believe others were running into this
issue and it was remedied by the work in
https://github.com/apache/nifi/pull/2273.

On Thu, Nov 16, 2017 at 9:46 PM, Mark Bean  wrote:

> I'm getting build failures when I attempt to build the latest master on one
> environment but not another. The failure was on macOS 10.12.6, maven 3.3.9,
> java 1.8.0_74. I agree, some of those could (and will be soon) updated.
>
> Yet, I was able to build successfully on an AWS EC platform earlier today.
> Exact specifics unavailable, but it was Linux-based, maven 3.5.0, java
> 1.8.0_152. So, it must be my environment. Given the error, perhaps
> something to do with available ports or port security?
>
> Curious if anyone else has seen this error:
>
> testRemotePortFingerprint(org.apache.nifi.fingerprint.
> FingerprintFactoryTest)
> Time elapsed: 0.02 sec  <<< FAILURE!
>
> org.junit.ComparisonFailure: expected: but
> was:
>
> at org.junit.Assert.assertEquals(Assert.java:115)
>
> at org.junit.Assert.assertEquals(Assert.java:144)
>
> at
> org.apache.nifi.fingerprint.FingerprintFactoryTest.
> testRemotePortFingerprint(FingerprintFactoryTest.java:317)
>

Re: Automated Testing of Flows

[Scott Aslan]

Just brain storming here

You could use travis or Jenkins to setup a job where you use mvn and npm to
build and start NiFi. Assuming the test template was available from within
the environment maybe you could use the protractor e2e testing suite to
script the clicks necessary to load the template and run the flow...

On Wed, Nov 15, 2017 at 8:07 PM, Peter Wicks (pwicks) 
wrote:

> I would like to find a way to automatically run integration/upgrade tests
> on flows, rather than just one or two processors like the mock framework
> supports. Preferably something Template driven, where I can provide a
> template xml file, and an accompanying config file to setup the test, then
> run the flow and examine the results.
>
> Has anyone worked on something like this?
>

build failure

[Mark Bean]

I'm getting build failures when I attempt to build the latest master on one
environment but not another. The failure was on macOS 10.12.6, maven 3.3.9,
java 1.8.0_74. I agree, some of those could (and will be soon) updated.

Yet, I was able to build successfully on an AWS EC platform earlier today.
Exact specifics unavailable, but it was Linux-based, maven 3.5.0, java
1.8.0_152. So, it must be my environment. Given the error, perhaps
something to do with available ports or port security?

Curious if anyone else has seen this error:

testRemotePortFingerprint(org.apache.nifi.fingerprint.FingerprintFactoryTest)
Time elapsed: 0.02 sec  <<< FAILURE!

org.junit.ComparisonFailure: expected: but
was:

at org.junit.Assert.assertEquals(Assert.java:115)

at org.junit.Assert.assertEquals(Assert.java:144)

at
org.apache.nifi.fingerprint.FingerprintFactoryTest.testRemotePortFingerprint(FingerprintFactoryTest.java:317)

Re: [DISCUSS] Apache NiFi MiNiFi C++ 0.3.0

[Marc]

Everyone,
  It looks as though the issues mentioned yesterday have been solved for
0.3.0. I apologize for being the long pole in addressing these issues in a
release that has seen some great work by a great group of contributors.

  A new ticket was created in response to an issue that helped expose
MINIFICPP-293 [0]. A fix for that ticket, MINIFICPP-308 [2], has been
submitted and is ready for review. I will try and go through the PR later
this evening or early tomorrow morning before a flight and merge it. That
will close out the release, unless anyone objects.

  I'm unavailable the next few days ( and will be in an isolated area
without internet) so the plan is to to tag and release Monday afternoon or
Tuesday morning, unless the release is needed sooner.

 Thanks,
 Marc
  [0]  https://issues.apache.org/jira/browse/MINIFICPP-293
  [1]  https://issues.apache.org/jira/browse/MINIFICPP-309
  [2] https://issues.apache.org/jira/browse/MINIFICPP-308

On Wed, Nov 15, 2017 at 8:33 AM, Marc  wrote:

> Everyone,
> I've been getting through some apache LDAP issues, hence the delay;
> however, we do have three remaining issues --
> https://issues.apache.org/jira/projects/MINIFICPP/versions/12341640.
>
>One is still under development. We can move it, but that would mean we
> can't compile in CENTOS 6. At this time my goal is to resolve the
> blockers/criticals since they would alienate whole groups of users. I've
> moved everything else from 0.3.0.
>
> On Thu, Nov 2, 2017 at 7:39 PM, Marc  wrote:
>
>> Aldrin,
>>   +1 to the idea of making a release.
>>
>>   Happy to try my hand at RM duties.
>>
>>   Can't promise I'll do well at it, but I can promise that when I screw
>> up, we'll all have a good laugh.
>>
>>
>>
>> On Wed, Nov 1, 2017 at 3:04 PM, Aldrin Piri  wrote:
>>
>>> Hey folks,
>>>
>>> We've had a fair amount of work and enhancements to the framework and
>>> extensions and seemed like a logical place to offer up a new release.
>>> Both
>>> Execute Script and HTTP Site to Site seem to be nearing their merge into
>>> master.  With a quick look through other open JIRAs it appears as some of
>>> the newer tickets starting progress might be extended efforts.
>>>
>>> Let me know if anyone has any thoughts or comments.  Otherwise, we can
>>> try
>>> to generate an RC and vote in the next week.  I am happy to act as RM if
>>> no
>>> one else is interested in doing so (likely have some docs I need to
>>> update
>>> to capture the process a bit better).
>>>
>>>
>>> --Aldrin
>>>
>>
>>
>

Run 1 instance of ExecuteStreamCommand constantly

[moe2017]

Hey,

I have python machine learning model being executed my an
ExecuteStreamCommand processor. The problem right now is that I need this
processor to be executed once and then consistently loop so that the model
can update itself when new data is passed to it from stdin.

I tried putting in a while(1) in my script and then sending the output to
stdout but NiFi just hangs and the processor won't take in the data from the
queue. Is NiFi not equipped to handle while loops? Or is there a NiFi way
that I can constantly run only one instance of a script during the entire
lifecycle of the pipeline?

Thanks



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: Issue with Executescript

[Vyshali]

I have subscribed.



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: Issue with Executescript

[Joe Witt]

It is awesome that you're using NiFi and engaging with the community.
However, please subscribe to the mailing list.  I will no longer
manually moderate these which mean they will not be seen unless you
subscribe or another moderator chooses to manually moderate.  If
you're unsure how to subscribe please let us know.  See here
https://nifi.apache.org/mailing_lists.html for more detail.

Thanks

On Thu, Nov 16, 2017 at 10:38 AM, Vyshali  wrote:
> Hi andy,
>
> I have successfully written the coding logic to do anonymization and was
> able to execute it without error.
> But I'm getting different results while running the same script on the same
> input in Nifi and as a normal python script.I'm not sure what is the
> problem.
>
> Sample dataset :
> Sharmila,sharmismi...@redmail.com,999-12-
> narasimha srinivasan,narasimma_sriniva...@gmail.com,222-26-789
> avyukt,vysh...@redmail.com,456-89-5678
>
> I have used seed functionality for maintaining consistency in the results of
> anonymization. So,I should get same results for multiple faker instances.I'm
> using the anonymizing code as normal python script and also executing in
> Nifi using executescript processor.
>
> When I run as python script,I'm getting the following output,
> Scott Bryan,bb...@yahoo.com,712-48-4862
> James Miranda,bradykait...@hotmail.com,446-57-4047
> James Jordan,fgar...@hotmail.com,887-47-4663
>
> When I execute the script in Nifi using executescript processor,I'm getting
> the following output:
> Andrew Simon,dncanrob...@hawkins.com,621-02-7781
> Gregory Grant,michell...@yahoo.com,709-80-9027
> Holly Nelson,bch...@yahoo.com,867-56-9800
>
> Whether problem will be due to Nifi using "jython" ? If so,how could we
> rectify across cross languages ?
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: Issue with Executescript

[Vyshali]

Hi andy,

I have successfully written the coding logic to do anonymization and was
able to execute it without error.
But I'm getting different results while running the same script on the same
input in Nifi and as a normal python script.I'm not sure what is the
problem.

Sample dataset :
Sharmila,sharmismi...@redmail.com,999-12-
narasimha srinivasan,narasimma_sriniva...@gmail.com,222-26-789
avyukt,vysh...@redmail.com,456-89-5678

I have used seed functionality for maintaining consistency in the results of
anonymization. So,I should get same results for multiple faker instances.I'm
using the anonymizing code as normal python script and also executing in
Nifi using executescript processor.

When I run as python script,I'm getting the following output,
Scott Bryan,bb...@yahoo.com,712-48-4862
James Miranda,bradykait...@hotmail.com,446-57-4047
James Jordan,fgar...@hotmail.com,887-47-4663

When I execute the script in Nifi using executescript processor,I'm getting
the following output:
Andrew Simon,dncanrob...@hawkins.com,621-02-7781
Gregory Grant,michell...@yahoo.com,709-80-9027
Holly Nelson,bch...@yahoo.com,867-56-9800

Whether problem will be due to Nifi using "jython" ? If so,how could we
rectify across cross languages ?



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/