Opinion wanted: Cyber orchestration "distro"

[Andre]

 Folks,

I have been privately working on a number of "processors" focused on
orchestration of cyber security related activities (eg update firewall
rules with data provided via an HTTP endpoint) etc.

While some of these tasks can be easily solved with generic NiFi components
or with little (or no custom processors at all) truth is that most security
practitioners just don't get it.

Result is that unless you show up with a processor called UpdateCiscoAcl
(random example), people's brains just melt.

I have been considering spinning up a separate project, based on a cut down
version of NiFi, that will employ the base framework towards this specific
use case by publishing specific processors that generally do not appeal to
the rest of the crowd.

My base rationale is the following:

- Reduce the need to add processors to the master tree and require people
to review processors that are of very limited use outside specific contexts.
- Improve overall user experience for this particular use case
- Reduce impact to the NiFi brand by the release of code that errr, may not
be up to the standards of my fellow committers ;-)

Given my position as a PMC member and profound respect to all of you, I
would like to reach out to the rest of the team for you overall thoughts
about this?

Looking forward to hearing from you.

Re: PutSyslog IP Address

[Andre]

John,

In addition to what Andrew said, if by ESM you refer to McAfee ESM, then
you need to be mindful it expects the system to send data in a particular
format and the data source to be configured on what the call a Forwarder.

Cheers



On Fri, Feb 23, 2018 at 11:49 PM, John Smith  wrote:

> Was just wondering how you (or anyone else) managed to solve this problem?
> We're doing something similar in that we're using Nifi to collect all our
> syslogs (using GetSysLog) and processing and forwarding it on to our ESM
> (using PutSysLog). The IP address which shows up in our ESM is the IP
> address of our Nifi box sending the syslog packets which is not ideal to
> say
> the least! My current thought is to write a custom processor but it would
> be
> good if I didn't have to do this!
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>

Re: [VOTE] Release Apache NiFi 1.5.0 (RC1)

[Andre]

Sending again as my previous vote seems to not have reached the mailing
list:

+1 (binding)


On Fri, Jan 12, 2018 at 3:36 PM, James Wing  wrote:

> +1 (binding).  Ran through the release helper, checked the signature,
> hashes, the license/notice changes, etc.  Tested the binary for a while.
>
> Great work!
>
> On Tue, Jan 9, 2018 at 2:19 AM, Joe Witt  wrote:
>
> > Hello,
> >
> > I am pleased to be calling this vote for the source release of Apache
> > NiFi nifi-1.5.0.
> >
> > The source zip, including signatures, digests, etc. can be found at:
> > https://repository.apache.org/content/repositories/orgapachenifi-1116
> >
> > The Git tag is nifi-1.5.0-RC1
> > The Git commit ID is 46d30c7e92f0ad034d9b35bf1d05c350ab5547ed
> > https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> > 46d30c7e92f0ad034d9b35bf1d05c350ab5547ed
> >
> > Checksums of nifi-1.5.0-source-release.zip:
> > MD5: 046f2dde4af592dd8c05e55c2bbb3c4f
> > SHA1: 63b9a68b9f89200fd31f5561956a15b45b1b9c8c
> > SHA256: 40b155c4911414907835f2eb0d5a4da798935f27f1e5134218d904fe6c942d13
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/joewitt.asc
> >
> > KEYS file available here:
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > 195 issues were closed/resolved for this release:
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > projectId=12316020&version=12341668
> >
> > Release note highlights can be found here:
> > https://cwiki.apache.org/confluence/display/NIFI/
> > Release+Notes#ReleaseNotes-Version1.5.0
> >
> > The vote will be open for 72 hours.
> > Please download the release candidate and evaluate the necessary items
> > including checking hashes, signatures, build
> > from source, and test.  The please vote:
> >
> > [ ] +1 Release this package as nifi-1.5.0
> > [ ] +0 no opinion
> > [ ] -1 Do not release this package because...
> >
>

Re: [DISCUSS] CI / Travis / Jenkins

[Andre]

Joe,

Thanks for that!

yes, you are correct. The builds occurred twice as the parallel build was
broken. :-)

Since we can now build and run tests in parallel with contrib-check
(YEAY) the dual build is no longer necessary.

THANK YOU for chasing this down. While travis is one of the drivers all the
developers will benefit from being able to run contrib-check in parallel
moving forward.

Cheers

On Thu, Dec 7, 2017 at 1:38 PM, Joe Witt  wrote:

> Team,
>
> Ok so finally some really solid news to share on the Travis-CI front.
> First, huge thanks to Aldrin for getting this started and folks like
> Andre and Pierre who have tweaked it to make it more usable as well.
> After a long run of it helping us out as we all know it went poorly
> with every build failing for what seemed like months.
>
> After some improvements and updates to our usage of maven which now
> means parallel builds with contrib check seem to be working and after
> going ruthless mode on hunting down unstable tests and either fixing
> them or making them integration-tests the build is far more stable.
> We all need to try and stay on top of that.  Today though i realized
> that our builds were happening twice and that appeared to be why it
> took roughly 50 minutes to finish, at best, and we'd timeout and fail.
>   So after adjusting our travis.yml we now only build once and the
> process takes about 25 mins so we're well within.
>
> Latest build on travis-ci: https://travis-ci.org/apache/
> nifi/builds/312629807
> Appveyor builds:
> https://ci.appveyor.com/project/ApacheSoftwareFoundation/nifi/
> build/1.0.0-SNAPSHOT-6649
>
> So we're heading in the right direction.  If it stays stable perhaps
> we could add openjdk builds as well.
>
> THanks
> Joe
>
> On Tue, Dec 5, 2017 at 4:11 PM, Joe Witt  wrote:
> > OK well things are looking pretty good.  The only obvious problem now
> > is that our builds take about 45-50 mins on travis-ci.org and the
> > build time limit is 50 mins [1] so some jobs get killed.
> >
> > Will look at areas we can avoid spending build time on at least in
> > travis-ci land.  Probably no great option but let's see.
> >
> > [1] https://docs.travis-ci.com/user/customizing-the-build#Build-Timeouts
> >
> > On Tue, Dec 5, 2017 at 2:56 PM, Joe Witt  wrote:
> >> Will try it out for PR https://github.com/apache/nifi/pull/2319 which
> >> is being built under
> >> https://travis-ci.org/apache/nifi/builds/312043710
> >>
> >> On Tue, Dec 5, 2017 at 2:51 PM, Joe Witt  wrote:
> >>> Andre
> >>>
> >>> Thanks - read through https://issues.apache.org/jira/browse/NIFI-1657
> >>> where this was discussed and where the relevant multi-env commit came
> >>> in.
> >>>
> >>> Seems like five environments may be too taxing based on the build
> >>> failures I'm observing.  I'll cut it down to three
> >>> FR
> >>> JP
> >>> US
> >>> For now.  We can evaluate if that helps at all and add more back if
> >>> things become stable.
> >>>
> >>> Thanks
> >>> Joe
> >>>
> >>> On Tue, Dec 5, 2017 at 12:20 AM, Andre  wrote:
> >>>> Joe,
> >>>>
> >>>> Glad to help! Few notes:
> >>>>
> >>>> If I recall correctly there was a reason we chose to add default and
> BR but
> >>>> to be honest I can't really remember what it was. I think it has to
> do with
> >>>> Time Zones + Locale issues and has helped detecting bizarre issues on
> time
> >>>> based junits (Matt B and Pierre may remember this).
> >>>>
> >>>> Regarding the rat check. The idea behind that was a fast failure in
> case of
> >>>> basic style violations, rather than wait until the end of the
> compilation.
> >>>> To be honest I don't know if this has worked as desired but should
> allow us
> >>>> to quickly identify validation errors which if I recall correctly
> were only
> >>>> detected at the end of contrib-check.
> >>>>
> >>>> And apologies for the anecdotal comments. I am away from my dev
> environment
> >>>> atm so I can't truly validate them.
> >>>>
> >>>>
> >>>> Kind regards
> >>>>
> >>>>
> >>>> On Tue, Dec 5, 2017 at 3:31 PM, Joe Witt  wrote:
> >>>>
> >>>>> Great news!  So for the first time in a long time we now have
> >>>>>

Re: [DISCUSS] CI / Travis / Jenkins

[Andre]

Joe,

Glad to help! Few notes:

If I recall correctly there was a reason we chose to add default and BR but
to be honest I can't really remember what it was. I think it has to do with
Time Zones + Locale issues and has helped detecting bizarre issues on time
based junits (Matt B and Pierre may remember this).

Regarding the rat check. The idea behind that was a fast failure in case of
basic style violations, rather than wait until the end of the compilation.
To be honest I don't know if this has worked as desired but should allow us
to quickly identify validation errors which if I recall correctly were only
detected at the end of contrib-check.

And apologies for the anecdotal comments. I am away from my dev environment
atm so I can't truly validate them.


Kind regards


On Tue, Dec 5, 2017 at 3:31 PM, Joe Witt  wrote:

> Great news!  So for the first time in a long time we now have
> travis-ci builds passing!
>
> I incorporated Dustin's PR which changed to the -Ddir-only instead of
> -P, added Andre's idea of dropping the -quiet flag, and dropped the
> number of builds in the config to a single parallel build with contrib
> check now that we're seeing those pass with rat/checkstyle.
>
> https://travis-ci.org/apache/nifi/builds/311660398
>
> A couple failed due to test failures and I filed JIRAs to convert
> these into integration tests or resolve.
>  -https://issues.apache.org/jira/browse/NIFI-4660,
> https://issues.apache.org/jira/browse/NIFI-4659
>
> One actually finished as you can see in its raw log but travis seems
> to have gotten confused.
>
> Two passed completely.  I think to reduce strain on Travis-CI
> infrastructure we should drop two of the environments.
>
> Current it is in .travis.yml
>
> env:
>   - USER_LANGUAGE=en USER_REGION=US'
>   - USER_LANGUAGE=fr USER_REGION=FR'
>   - USER_LANGUAGE=ja USER_REGION=JP'
>   - USER_LANGUAGE=pt USER_REGION=BR'
>   - USER_LANGUAGE=default USER_REGION=default
>
> I think we should drop it to
>
> env:
>   - USER_LANGUAGE=en USER_REGION=US'
>   - USER_LANGUAGE=fr USER_REGION=FR'
>   - USER_LANGUAGE=ja USER_REGION=JP'
>
> If no objections i'll do that soon.  But, good news is the builds are
> coming back to life on Travis-CI and will help streamline review
> cycles again!
>
> Thanks
>
> On Mon, Dec 4, 2017 at 8:29 PM, Joe Witt  wrote:
> > nope. will take a look at this tonight though.
> >
> > On Dec 4, 2017 8:09 PM, "Andre"  wrote:
> >>
> >> Joe & Joey,
> >>
> >> I believe setting the maven compilation job to noisy - instead of the
> >> current quiet setting - should help solving the issue.
> >>
> >> Have we tried that?
> >>
> >> Cheers
> >>
> >>
> >> On 5 Dec 2017 6:26 AM, "Joe Witt"  wrote:
> >>
> >> I agree this would be extremely nice to get back on track.  The
> >> changes made last night/today to the poms do appear to mean that
> >> parallel builds with contrib-check are working.  Perhaps that helps us
> >> a little with travis (or not).  I have reviewed a couple PRs though
> >> recently that did not even compile much less have clean contrib-checks
> >> so it is really nice to have Travis being more reliable.  Does anyone
> >> have any sense of the current reasons for issues?  When I've looked
> >> the errors made no sense at all.
> >>
> >> On Mon, Dec 4, 2017 at 2:21 PM, Joey Frazee 
> >> wrote:
> >> > I’m sure everyone has noticed that Travis CI fails, incorrectly, more
> >> than it succeeds, often due to timeouts and not b/c of the incorrectness
> >> of
> >> a commit or PR.
> >> >
> >> > This has been discussed previously, but it’s carried on, and become a
> >> > low
> >> information signal about the PRs, which has two big impacts: (1) it’s
> >> ignored by experienced contributors and reviewers, and (2) it’s
> confusing
> >> or misleading to new contributors.
> >> >
> >> > So, we really need to find a solution. I can think of a few:
> >> >
> >> > 1. Continue to push on INFRA to setup Jenkins for NiFi and its
> >> sub-projects.
> >> >
> >> > 2. Implement some kind of quick-test profile and shell script that
> >> > checks
> >> the most important things along with the subdirectories affected by the
> >> PR,
> >> and continue to use Travis CI.
> >> >
> >> > 3. Use some other service like Circle CI or Codeship, which probably
> >> isn’t quite 

Re: [DISCUSS] CI / Travis / Jenkins

[Andre]

Joe & Joey,

I believe setting the maven compilation job to noisy - instead of the
current quiet setting - should help solving the issue.

Have we tried that?

Cheers


On 5 Dec 2017 6:26 AM, "Joe Witt"  wrote:

I agree this would be extremely nice to get back on track.  The
changes made last night/today to the poms do appear to mean that
parallel builds with contrib-check are working.  Perhaps that helps us
a little with travis (or not).  I have reviewed a couple PRs though
recently that did not even compile much less have clean contrib-checks
so it is really nice to have Travis being more reliable.  Does anyone
have any sense of the current reasons for issues?  When I've looked
the errors made no sense at all.

On Mon, Dec 4, 2017 at 2:21 PM, Joey Frazee  wrote:
> I’m sure everyone has noticed that Travis CI fails, incorrectly, more
than it succeeds, often due to timeouts and not b/c of the incorrectness of
a commit or PR.
>
> This has been discussed previously, but it’s carried on, and become a low
information signal about the PRs, which has two big impacts: (1) it’s
ignored by experienced contributors and reviewers, and (2) it’s confusing
or misleading to new contributors.
>
> So, we really need to find a solution. I can think of a few:
>
> 1. Continue to push on INFRA to setup Jenkins for NiFi and its
sub-projects.
>
> 2. Implement some kind of quick-test profile and shell script that checks
the most important things along with the subdirectories affected by the PR,
and continue to use Travis CI.
>
> 3. Use some other service like Circle CI or Codeship, which probably
isn’t quite what ASF wants but it might make the CI more useful (it also
might not).
>
> 4. Find a sponsor to support a more premium tier of Travis CI (or equiv.)
so the build has enough resources to to succeed. This too probably isn’t
preferable but I’m sure we can find a precedent.
>
> I’m partial to pursuing (1) and (2) together because (1) would give us a
long term solution and (2) would have some value for local builds (no need
to run the full build) as well as making Travis CI tell us something. The
first should be pretty low effort. The second will be labor intensive I
think — to identify what counts as quick and change the poms — so it can’t
be the answer on its own unless we want to wait longer to see Travis CI
become informative.
>
> What do the rest of you think?
>
> -joey

Re: The mystery of ListSFTP that stops working

[Andre]

Hey Mark,

Changing the scheduling doesn't seem to make a difference.

I have noticed some instances where a nifi.sh dump causes a switch of the
primary node and that unleashes the stuck processor but still not clear why
it happens.

I took a thread dump of the primary node while the thing was happening.
Will upload to gist soon.

Cheers



On Thu, Nov 16, 2017 at 2:07 AM, Mark Payne  wrote:

> Andre,
>
> So I am guessing that backpressure is not an issue then if you're not
> seeing it run :)
> Have you tried reducing the scheduling period from 5 mins to something
> like 5 seconds?
> Of course, you may not want to actually be running it every 5 seconds in a
> production environment,
> but I am curious if it would cause it to start running or not...
>
> > On Nov 15, 2017, at 10:02 AM, Andre  wrote:
> >
> > Mark,
> >
> > Timer driven, Primary Node, 5 min
> > Yield is set to 1 sec
> > Backpressure = 10k flows or 1GB
> > nifi.administrative.yield.duration=30 sec
> >
> > Cheers
> >
> >
> > On Thu, Nov 16, 2017 at 1:32 AM, Mark Payne 
> wrote:
> >
> >> Hey Andre,
> >>
> >> I have not seen this personally. Can you share how you have the
> Scheduling
> >> Tab configured?
> >> Is it set to Timer-Driven with a period of "0 secs"? Also, what is the
> >> Yield Duration set to?
> >> Is there any backpressure configured on any of the outbound connections?
> >> Additionally, what is the value of the "nifi.administrative.yield.
> duration"
> >> property in nifi.properties?
> >>
> >> Sorry - I know that's a barrage of questions. Hopefully it's something
> >> easy that's just being overlooked, though :)
> >>
> >> -Mark
> >>
> >>> On Nov 15, 2017, at 4:58 AM, Andre  wrote:
> >>>
> >>> Folks,
> >>>
> >>> Has anyone ever seen a situation where ListSFTP simply stops working?
> >>>
> >>> I have observed a few occurrences of seems to be some weird bug.
> >>>
> >>> Symptoms are:
> >>>
> >>> - Processor looks healthy (i.e. no bulletins)
> >>> - State is stalled (i.e. no changes to new files or timestamps)
> >>> - No signs of a stuck thread (i.e. processor doesn't display a thread
> >> count)
> >>> - UI displays processor "Task count = 0/0"
> >>> - Neither stopping > starting nor stopping > disabling > enabling >
> >>> starting processor seems to make a difference.
> >>>
> >>> Has anyone seen this?
> >>>
> >>> Cheers
> >>
> >>
>
>

Re: The mystery of ListSFTP that stops working

[Andre]

Mark,

Timer driven, Primary Node, 5 min
Yield is set to 1 sec
Backpressure = 10k flows or 1GB
nifi.administrative.yield.duration=30 sec

Cheers


On Thu, Nov 16, 2017 at 1:32 AM, Mark Payne  wrote:

> Hey Andre,
>
> I have not seen this personally. Can you share how you have the Scheduling
> Tab configured?
> Is it set to Timer-Driven with a period of "0 secs"? Also, what is the
> Yield Duration set to?
> Is there any backpressure configured on any of the outbound connections?
> Additionally, what is the value of the "nifi.administrative.yield.duration"
> property in nifi.properties?
>
> Sorry - I know that's a barrage of questions. Hopefully it's something
> easy that's just being overlooked, though :)
>
> -Mark
>
> > On Nov 15, 2017, at 4:58 AM, Andre  wrote:
> >
> > Folks,
> >
> > Has anyone ever seen a situation where ListSFTP simply stops working?
> >
> > I have observed a few occurrences of seems to be some weird bug.
> >
> > Symptoms are:
> >
> > - Processor looks healthy (i.e. no bulletins)
> > - State is stalled (i.e. no changes to new files or timestamps)
> > - No signs of a stuck thread (i.e. processor doesn't display a thread
> count)
> > - UI displays processor "Task count = 0/0"
> > - Neither stopping > starting nor stopping > disabling > enabling >
> > starting processor seems to make a difference.
> >
> > Has anyone seen this?
> >
> > Cheers
>
>

The mystery of ListSFTP that stops working

[Andre]

Folks,

Has anyone ever seen a situation where ListSFTP simply stops working?

I have observed a few occurrences of seems to be some weird bug.

Symptoms are:

- Processor looks healthy (i.e. no bulletins)
- State is stalled (i.e. no changes to new files or timestamps)
- No signs of a stuck thread (i.e. processor doesn't display a thread count)
- UI displays processor "Task count = 0/0"
- Neither stopping > starting nor stopping > disabling > enabling >
starting processor seems to make a difference.

Has anyone seen this?

Cheers

Re: Did 1.4.0 introduce a "breaking change" ?

[Andre]

Matt,

Yes. It seems to be the case. Reason being the Config Mgmt techniques used
to populate the files are different.

Cheers

On Wed, Nov 1, 2017 at 12:07 AM, Matt Gilman 
wrote:

> Andre,
>
> Based on the error, it looks like you've retained your existing
> authorizers.xml but upgraded the nifi.security.user.authorizer property in
> your nifi.properties. These two are related. The authorizers.xml
> file defines the available Authorizers (and now UserGroupProviders and
> AccessPolicyProviders). The nifi.security.user.authorizer property in
> nifi.properties defines which authorizer to use (via the identifier) from
> the authorizers.xml file. If you want to retain your existing
> authorizers.xml you'll want to keep that property unchanged as well.
>
> I'll add a note in the migration guide to make sure this is clear.
>
> Thanks
>
> Matt
>
> On Tue, Oct 31, 2017 at 12:05 AM, Andre  wrote:
>
> > Matt,
> >
> > I did some investigation and it seems like if you simply copy the old
> > authorizers.xml into the new setup NiFi fails with
> >
> > ERROR [main] o.s.web.context.ContextLoader Context initialization failed
> > org.springframework.beans.factory.BeanCreationException: Error creating
> > bean with name 'niFiWebApiSecurityConfiguration': Injection of autowired
> > dependencies failed; nested exception is
> > org.springframework.beans.factory.BeanCreationException: Could not
> > autowire
> > method: public void
> > org.apache.nifi.web.NiFiWebApiSecurityConfiguration.
> > setOtpAuthenticationProvider(org.apache.nifi.web.security.
> > otp.OtpAuthenticationProvider);
> > nested exception is
> > org.springframework.beans.factory.BeanCreationException: Error creating
> > bean with name 'otpAuthenticationProvider' defined in class path resource
> > [nifi-web-security-context.xml]: Cannot resolve reference to bean
> > 'authorizer' while setting constructor argument; nested exception is
> > org.springframework.beans.factory.BeanCreationException: Error creating
> > bean with name 'authorizer': FactoryBean threw exception on object
> > creation; nested exception is java.lang.Exception: The specified
> authorizer
> > 'managed-authorizer' could not be found.
> >
> > And similar errors for JWT as well.
> >
> > Given that populating the new authorizers.xml with the 
> element
> > from the old authorizers.xml seems to solve the issue, seems to me like
> we
> > expect the file to contain references to the new authorizers syntax
> > despite using the legacy syntax?
> >
> > Doesn't seem to be a show stopper but I reckon we should document it.
> >
> > Cheers
> >
> >
> >
> > On Tue, Oct 31, 2017 at 1:43 PM, Matt Gilman 
> > wrote:
> >
> > > Andre,
> > >
> > > While 1.4.0 introduces a more granular authorizers configuration, the
> > > existing 1.3.0 configurations should still be valid. What was breaking
> > for
> > > you?
> > >
> > > Matt
> > >
> > > Sent from my iPhone
> > >
> > > > On Oct 30, 2017, at 10:24 PM, Andre  wrote:
> > > >
> > > > Folks,
> > > >
> > > > I was looking at the upgrade process from 1.3.x to 1.4.0 and it seems
> > to
> > > me
> > > > we introduced a breaking change around the authorizations?
> > > >
> > > > When I look at the 1.4.0 authorizers.xml and its version 1.3.0
> there's
> > a
> > > > massive discrepancy on the XML tree and the existing 1.3.x version
> does
> > > not
> > > > seem to work out of the box on 1.4.0 ?
> > > >
> > > > Looking at the docs I couldn't find evidence of the migration being
> > fully
> > > > documented? I suspect users may be able to find their way by
> adjusting
> > > the
> > > > instructions for the 0.x upgrade, but shouldn't we have clear
> > > instructions
> > > > on the upgrade from 1.3.x to 1.4.0?
> > > >
> > > > Cheers
> > >
> >
>

Re: Did 1.4.0 introduce a "breaking change" ?

[Andre]

Matt,

I did some investigation and it seems like if you simply copy the old
authorizers.xml into the new setup NiFi fails with

ERROR [main] o.s.web.context.ContextLoader Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'niFiWebApiSecurityConfiguration': Injection of autowired
dependencies failed; nested exception is
org.springframework.beans.factory.BeanCreationException: Could not autowire
method: public void
org.apache.nifi.web.NiFiWebApiSecurityConfiguration.setOtpAuthenticationProvider(org.apache.nifi.web.security.otp.OtpAuthenticationProvider);
nested exception is
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'otpAuthenticationProvider' defined in class path resource
[nifi-web-security-context.xml]: Cannot resolve reference to bean
'authorizer' while setting constructor argument; nested exception is
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'authorizer': FactoryBean threw exception on object
creation; nested exception is java.lang.Exception: The specified authorizer
'managed-authorizer' could not be found.

And similar errors for JWT as well.

Given that populating the new authorizers.xml with the  element
from the old authorizers.xml seems to solve the issue, seems to me like we
expect the file to contain references to the new authorizers syntax
despite using the legacy syntax?

Doesn't seem to be a show stopper but I reckon we should document it.

Cheers



On Tue, Oct 31, 2017 at 1:43 PM, Matt Gilman 
wrote:

> Andre,
>
> While 1.4.0 introduces a more granular authorizers configuration, the
> existing 1.3.0 configurations should still be valid. What was breaking for
> you?
>
> Matt
>
> Sent from my iPhone
>
> > On Oct 30, 2017, at 10:24 PM, Andre  wrote:
> >
> > Folks,
> >
> > I was looking at the upgrade process from 1.3.x to 1.4.0 and it seems to
> me
> > we introduced a breaking change around the authorizations?
> >
> > When I look at the 1.4.0 authorizers.xml and its version 1.3.0 there's a
> > massive discrepancy on the XML tree and the existing 1.3.x version does
> not
> > seem to work out of the box on 1.4.0 ?
> >
> > Looking at the docs I couldn't find evidence of the migration being fully
> > documented? I suspect users may be able to find their way by adjusting
> the
> > instructions for the 0.x upgrade, but shouldn't we have clear
> instructions
> > on the upgrade from 1.3.x to 1.4.0?
> >
> > Cheers
>

Did 1.4.0 introduce a "breaking change" ?

[Andre]

Folks,

I was looking at the upgrade process from 1.3.x to 1.4.0 and it seems to me
we introduced a breaking change around the authorizations?

When I look at the 1.4.0 authorizers.xml and its version 1.3.0 there's a
massive discrepancy on the XML tree and the existing 1.3.x version does not
seem to work out of the box on 1.4.0 ?

Looking at the docs I couldn't find evidence of the migration being fully
documented? I suspect users may be able to find their way by adjusting the
instructions for the 0.x upgrade, but shouldn't we have clear instructions
on the upgrade from 1.3.x to 1.4.0?

Cheers

[Discuss] DeleteSFTP

[Andre]

Folks,

I was recently asked by a user if we have a DeleteSFTP processor to handle
file deletion AFTER some logic has been performed.

I noticed we don't have a processor covering that, and while ExecuteScript
could be used to make this happen I wondered if it we should support that?

The logic is simple:

ListSFTP -> FetchSFTP  - (if success) -> DoSomething - (if success) ->
DeleteSFTP


I think this pattern would fit those who are just too cautious to trust the
"Delete file" "Completion strategy" available on FetchSFTP.  The reason for
caution is the assumption that although NiFi queues are persistent, things
can still go wrong between the file being sourced and the "DoSomething"
success obtained.

What do you think?

Re: Want to contribute Jira[NIFI-4360]

[Andre]

Milan,

Would you be able to comment on the differences between NIFI-4360 and
NIFI-1922 ?

Cheers

On Thu, Sep 14, 2017 at 1:00 PM, Milan Chandna <
milan.chan...@microsoft.com.invalid> wrote:

> Hi,
>
> Can admins please make me a contributor so that I can assign
> Jira[NIFI-4360] (that I
> raised) to myself.
>
> Regards,
> Milan.
>

Re: nifi processor for IPFIX

[Andre]

Chris,

I was planning to address the netflow processor sometime in August once I
finish the work I am doing around consuming TAXII endpoints but I am also
happy for someone to give it a go...

The plan was to process IPFIX (as this is the open standard) and later on,
depending on documentation available process netflow.


Cheers

On Wed, Jul 26, 2017 at 5:09 PM, Chris Herssens 
wrote:

> Hello Joe,
>
> Thanks for your answer. I found a Jira ticket about implementation of a
> netflow procerssor.
> Do you know what the status is ?   Since Netflow and IPFIX are important
> protocols for network monitoring, It would be nice to have such kind
> of processors.
> Where can I find more information about the ListenTCPRecord ?
>
> Regards,
>
> Chris
>
> On Tue, Jul 25, 2017 at 4:24 PM, Joe Witt  wrote:
>
> > Chris,
> >
> > There are no plans that I am aware of.  We'd need to have build a
> > ListenUDPRecord processor and we'd need an IPFIXRecordReader.  This
> > would be pretty slick and quite fast.  I *think* Bryan Bende was
> > working on a ListenTCPRecord so maybe this could be tied into that.
> >
> > Thanks
> >
> > On Tue, Jul 25, 2017 at 9:06 AM, Chris Herssens
> >  wrote:
> > > Hello All,
> > >
> > > Are there plans to implement an IPFIX collector.
> > > The processor should listen on an UDP port, parse the content and
> convert
> > > it to AVRO or JSON
> > >
> > > Regards,
> > >
> > > Chris
> >
>

Connection - Status History - Expired Flows

[Andre]

Folks,

Flowfile expiration is a great feature in NiFi however I noticed a small
oddity in how one can detect expiration of Flowfiles.

>From what I understand, currently, the only way possible to do that is
searching the provenance events for the EXPIRE event type.

I've searched JIRA and noticed NIFI-620 flags this out. Is this something
we should work on or is the current approach (via reporting) the preferred
choice?

Could anyone express the advantages of one versus the other?

Cheers

Re: Filesystem based Schema Registry, does it make sense?

[Andre]

Joe,

At least in my case, I would be happy to control versioning outside nifi
(eg via git).

I would suspect that by adopting an approach like this versioning would be
handled pretty much like the AvroSchemaRegistry?

I assumed that with AvroSchemaRegistry you either name your schema with
versioning in mind (ie mySchemav1)  or you need to overwrite the schema
upon change? Am I missing something?

Cheers

On 6 Jul 2017 1:59 AM, "Joe Witt"  wrote:

I think it does make sense and someone at a meetup asked a similar
question.  There are some things to be considered like how does one
annotate the version of a schema, the name, etc.. when all they are
providing are files in a directory?  How can they support multiple versions
of a given schema (or maybe they just dont in this approach)?  But there is
no question that being able to just push an avsc file into a directory and
then have it be useable in the flow could be helpful.

On Jul 5, 2017 9:00 AM, "Andre"  wrote:

dev,

As I continue to explore the Record based processors I got myself wondering:

Does it make sense to have a file-system based schema registry?

Idea would be creating something like AvroSchemaRegistry but instead of the
adding each schema as a controller service property, we would have a
property pointing to a directory.

Each avsc file within that directory would then be validated with the root
"name" within the Avro schema used as the schema name (i.e. the equivalent
to AvroSchemaRegistry property name).

The rationale is that while the Hortonworks and Avro Schema Registries
work, I reckon one is sort of overkill for edge/DMZ NiFi deployments and
the other is painful to update in case of multiple NiFi clusters.

Having a file based registry with inotify or something of sort would be
great for the folks already using external configuration management.


What do you think?

Filesystem based Schema Registry, does it make sense?

[Andre]

dev,

As I continue to explore the Record based processors I got myself wondering:

Does it make sense to have a file-system based schema registry?

Idea would be creating something like AvroSchemaRegistry but instead of the
adding each schema as a controller service property, we would have a
property pointing to a directory.

Each avsc file within that directory would then be validated with the root
"name" within the Avro schema used as the schema name (i.e. the equivalent
to AvroSchemaRegistry property name).

The rationale is that while the Hortonworks and Avro Schema Registries
work, I reckon one is sort of overkill for edge/DMZ NiFi deployments and
the other is painful to update in case of multiple NiFi clusters.

Having a file based registry with inotify or something of sort would be
great for the folks already using external configuration management.


What do you think?

Re: AvroSchemaRegistry doesn't enjoy copy and paste?

[Andre]

Andrew,

Your are correct The JSON's must be different... but I have not even gone
to that level yet! NiFi simply failed to validate the second schema (with
nested records)

The thing that was not clear to my is why the Avro Registry Service would
not validate the second schema, event after minifying it and confirming
both single quotes and double quotes are pure ASCII characters.

It turned out to be the issue is fixed by NIFI-4029

Are we targeting releasing 1.3.1 by any chance? 😎

Cheers


PS-I wasn't familiar with avro-tools random, so thank for that! 😀


On Fri, Jun 23, 2017 at 2:57 PM, Andrew Psaltis 
wrote:

> Andre,
> I may be totally off-base here as it relates to the record related
> processing in NiFi. From a pure Avro viewpoint the same JSON will not work
> between those two schemas that you provided. The second one has a nested
> header record and the first does not. I used the java AVRO tools to do the
> following:
>
>
>1. Generate sample Avro for schema 1
>
>  java -jar avro-tools-1.7.7.jar random --count 1 --schema-file schema1.asvc
> test1.avro
>
>
> 2.  Convert binary Avro to JSON
>
>
> java -jar avro-tools-1.7.7.jar tojson  test1.avro
>
>
> This produces JSON like such:
>
>
> {"version":297340384,"deviceVendor":null}
>
>
> 3. Generate sample Avro for schema 2
>
> java -jar avro-tools-1.7.7.jar random --count 1 --schema-file schema2.asvc
> test2.avro
>
>
> 4. Convert binary Avro to JSON
>
>
> java -jar avro-tools-1.7.7.jar tojson  test2.avro
>
>
> This produces JSON like such:
>
>
> {"header":{"version":-492928400,"deviceVendor":{"float":0.59934044}}}
>
>
> To then have valid JSON with the second schema, you would represent it as:
>
> {"header":{"version":-492928400,"deviceVendor":null}}
>
>
> If you try and generate Avro with JSON from step 2 above
> ({"version":297340384,"deviceVendor":null}) using the second schema:
>
>
> java -jar avro-tools-1.7.7.jar fromjson --schema-file schema2.asvc
> test1.json
>
> It will throw an exception:
>
>
> avro.codenull3+�MW���o���c��Exception in thread "main"
> org.apache.avro.AvroTypeException: Expected field name not found: header
>
>
> Hopefully this did not totally muddy the water.
>
> Thanks,
> Andrew
>
>
>
> On Thu, Jun 22, 2017 at 11:00 PM, Andre  wrote:
>
> > All,
> >
> > So it turns out the reason for the validation failure seems to be arising
> > here:
> >
> > https://github.com/apache/nifi/blob/b1901d5fe0bf87be3dcce144f13b74
> > eb995be168/nifi-commons/nifi-record/src/main/java/org/
> > apache/nifi/serialization/record/RecordField.java#L44
> >
> > When using the following schema:
> >
> > {
> > "type":"record",
> > "name":"header",
> > "fields":[
> >   {
> > "name":"version",
> > "type":"int",
> > "doc":"The CEF version extracted from (CEF:0) where 0 is the
> > version"
> >   },
> >   {
> > "name":"deviceVendor",
> > "type":[
> >   "null",
> >   "float"
> > ],
> > "default":null
> >   }
> > ]
> >
> > }
> >
> > The float field with default Null will parse correctly.
> >
> > However
> >
> > When using this:
> >
> > {
> >   "name":"nifiCommonEventFormat",
> >   "namespace":"com.fluenda.SecuritySchemas.cefRev23",
> >   "type":"record",
> >   "fields":[
> > {
> >   "name":"header",
> >   "type":{
> > "type":"record",
> > "name":"header",
> > "fields":[
> >   {
> >     "name":"version",
> > "type":"int",
> > "doc":"The CEF version extracted from (CEF:0) where 0 is the
> > version"
> >   },
> >   {
> > "name":"deviceVendor",
> > "type":[
> >   "null",
> >   "float"
> > ],
> > "default":null
> >   }
> >

Re: AvroSchemaRegistry doesn't enjoy copy and paste?

[Andre]

All,

So it turns out the reason for the validation failure seems to be arising
here:

https://github.com/apache/nifi/blob/b1901d5fe0bf87be3dcce144f13b74eb995be168/nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/RecordField.java#L44

When using the following schema:

{
"type":"record",
"name":"header",
"fields":[
  {
"name":"version",
"type":"int",
"doc":"The CEF version extracted from (CEF:0) where 0 is the
version"
  },
  {
"name":"deviceVendor",
"type":[
  "null",
  "float"
],
"default":null
  }
]

}

The float field with default Null will parse correctly.

However

When using this:

{
  "name":"nifiCommonEventFormat",
  "namespace":"com.fluenda.SecuritySchemas.cefRev23",
  "type":"record",
  "fields":[
{
  "name":"header",
  "type":{
"type":"record",
"name":"header",
"fields":[
  {
"name":"version",
"type":"int",
"doc":"The CEF version extracted from (CEF:0) where 0 is the
version"
  },
  {
"name":"deviceVendor",
"type":[
  "null",
  "float"
],
"default":null
  }
]
  }
}
  ]
}

It fails.

intelliJ seems to suggest an NPE is the underlying cause?

[image: Inline image 1]


What puzzles me, is that I tested with python and the schema seems valid?

$ ./test.py
{u'header': {u'version': 0, u'deviceVendor': 1.2345000505447388}}
{u'header': {u'version': 0, u'deviceVendor': None}}


The test.py script can be found here:

https://gist.github.com/trixpan/4d190bef83712001857548b63c1d995a


Cheers









On Fri, Jun 23, 2017 at 12:20 AM, Andre  wrote:

> Joey,
>
> Using Chrome on Windows. Tried with Firefox and faced the same issue.
>
> On Fri, Jun 23, 2017 at 12:14 AM, Joey Frazee 
> wrote:
>
>> Andre, if there are any line breaks in the schema and leading spaces on
>> those new lines, then this occurs. So if you minify the avsc or remove the
>> leading spaces, all should be good.
>>
>> Will open a JIRA on this since, including myself, I think you’re the
>> third to see this. Any chance you’re using Safari?
>>
>> > On Jun 22, 2017, at 8:53 AM, Andrew Grande  wrote:
>> >
>> > Definitely something is auto replacing quotes, I can confirm pasting
>> worked
>> > fine before from a programmer's editor.
>> >
>> > Andrew
>> >
>> > On Thu, Jun 22, 2017, 9:06 AM Mark Payne  wrote:
>> >
>> >> Andre,
>> >>
>> >> I've not seen this personally. I just clicked on the link you sent,
>> copied
>> >> the schema,
>> >> and pasted it in, and it did not have any problems. What application
>> are
>> >> you copying
>> >> the text from? I've certainly seen that some applications (specifically
>> >> Microsoft Outlook
>> >> and Office) love to take double-quotes and change them into other
>> >> characters so that
>> >> they look nicer. But if you then copy that and paste it, it is not
>> pasting
>> >> a double-quote but
>> >> some other unicode character.
>> >>
>> >> Would recommend you open the below link in Chrome and copy from there
>> and
>> >> see if
>> >> that works?
>> >>
>> >> Thanks
>> >> -Mark
>> >>
>> >>
>> >>
>> >>> On Jun 22, 2017, at 8:56 AM, Andre  wrote:
>> >>>
>> >>> All,
>> >>>
>> >>> I was playing with the AvroSchemaRegistry and noticed it seems to not
>> >> play
>> >>> ball when the DFM pastes the schema into the dynamic property value.
>> >>>
>> >>> To test it I basically copied the demo schema from Mark's blog post[1]
>> >> and
>> >>> pasted into a NiFi 1.3.0 instance. To my surprise the controller would
>> >> not
>> >>> validate, instead it displayed:
>> >>>
>> >>> "was expecting double-quote to start field name"
>> >>>
>> >>> I also faced similar errors using the following schema:
>> >>>
>> >>>
>> >> https://github.com/fluenda/SecuritySchemas/blob/master/CEFRe
>> v23/cefRev23_nifi.avsc
>> >>>
>> >>> Has anyone else seen this?
>> >>>
>> >>> Cheers
>> >>
>> >>
>>
>>
>

Re: AvroSchemaRegistry doesn't enjoy copy and paste?

[Andre]

Joey,

Using Chrome on Windows. Tried with Firefox and faced the same issue.

On Fri, Jun 23, 2017 at 12:14 AM, Joey Frazee 
wrote:

> Andre, if there are any line breaks in the schema and leading spaces on
> those new lines, then this occurs. So if you minify the avsc or remove the
> leading spaces, all should be good.
>
> Will open a JIRA on this since, including myself, I think you’re the third
> to see this. Any chance you’re using Safari?
>
> > On Jun 22, 2017, at 8:53 AM, Andrew Grande  wrote:
> >
> > Definitely something is auto replacing quotes, I can confirm pasting
> worked
> > fine before from a programmer's editor.
> >
> > Andrew
> >
> > On Thu, Jun 22, 2017, 9:06 AM Mark Payne  wrote:
> >
> >> Andre,
> >>
> >> I've not seen this personally. I just clicked on the link you sent,
> copied
> >> the schema,
> >> and pasted it in, and it did not have any problems. What application are
> >> you copying
> >> the text from? I've certainly seen that some applications (specifically
> >> Microsoft Outlook
> >> and Office) love to take double-quotes and change them into other
> >> characters so that
> >> they look nicer. But if you then copy that and paste it, it is not
> pasting
> >> a double-quote but
> >> some other unicode character.
> >>
> >> Would recommend you open the below link in Chrome and copy from there
> and
> >> see if
> >> that works?
> >>
> >> Thanks
> >> -Mark
> >>
> >>
> >>
> >>> On Jun 22, 2017, at 8:56 AM, Andre  wrote:
> >>>
> >>> All,
> >>>
> >>> I was playing with the AvroSchemaRegistry and noticed it seems to not
> >> play
> >>> ball when the DFM pastes the schema into the dynamic property value.
> >>>
> >>> To test it I basically copied the demo schema from Mark's blog post[1]
> >> and
> >>> pasted into a NiFi 1.3.0 instance. To my surprise the controller would
> >> not
> >>> validate, instead it displayed:
> >>>
> >>> "was expecting double-quote to start field name"
> >>>
> >>> I also faced similar errors using the following schema:
> >>>
> >>>
> >> https://github.com/fluenda/SecuritySchemas/blob/master/
> CEFRev23/cefRev23_nifi.avsc
> >>>
> >>> Has anyone else seen this?
> >>>
> >>> Cheers
> >>
> >>
>
>

AvroSchemaRegistry doesn't enjoy copy and paste?

[Andre]

All,

I was playing with the AvroSchemaRegistry and noticed it seems to not play
ball when the DFM pastes the schema into the dynamic property value.

To test it I basically copied the demo schema from Mark's blog post[1]  and
pasted into a NiFi 1.3.0 instance. To my surprise the controller would not
validate, instead it displayed:

"was expecting double-quote to start field name"

I also faced similar errors using the following schema:

https://github.com/fluenda/SecuritySchemas/blob/master/CEFRev23/cefRev23_nifi.avsc

Has anyone else seen this?

Cheers

Re: LookupAttribute + CSV Lookup handling of non-matches

[Andre]

Joey,

Thanks for the reply.

Yeah, I probably would get to that property soon or later but I reckon the
property description should be revisited. I think we should make it quite
explicit that setting this to true will result in unmatched to stop working
as expected.

On Thu, Jun 22, 2017 at 2:43 PM, Joey Frazee  wrote:

> Andre, there's a flag for "Include Empty Values". If set to false then the
> effect is that non-matches result in a transfer to unmatched. The default
> is true so I'd expect to see what you're seeing.
>
> The description for that property isn't good enough though. I don't know
> how you would have known this now that I'm looking at the docs again.
>
> -joey
>
> > On Jun 21, 2017, at 11:36 PM, Andre  wrote:
> >
> > Hi all,
> >
> > I have been playing with the lookup attribute and noticed a behavior that
> > seems counter intuitive to me:
> >
> > When using the SimpleCSVLookupService I noticed that whenever the service
> > fails to find a match to a particular value, the LookupAttribute
> processor
> > still populated the respective dynamic property with "null" and routed
> the
> > flowfile to match instead of leaving the field untouched and transferring
> > the flow to unmatched.
> >
> > Is this expected behavior? If so, what is the use of unmatched?
> >
> > Kind regards
>

LookupAttribute + CSV Lookup handling of non-matches

[Andre]

Hi all,

I have been playing with the lookup attribute and noticed a behavior that
seems counter intuitive to me:

When using the SimpleCSVLookupService I noticed that whenever the service
fails to find a match to a particular value, the LookupAttribute processor
still populated the respective dynamic property with "null" and routed the
flowfile to match instead of leaving the field untouched and transferring
the flow to unmatched.

Is this expected behavior? If so, what is the use of unmatched?

Kind regards

Re: [VOTE] Release Apache NiFi 1.3.0

[Andre]

+1 binding

On Wed, Jun 7, 2017 at 1:08 PM, Aldrin Piri  wrote:

> +1, binding
>
> Hashes and signature looks good
> Build and contrib check was clean
> README, L&N look good
> Verified some simple flows
>
>
> Environment:
> Apache Maven 3.3.1 (cab6659f9874fa96462afef40fcf6bc033d58c1c;
> 2015-03-13T16:10:27-04:00)
> Maven home: /usr/local/maven
> Java version: 1.8.0_121, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-0.b13.el7_3.
> x86_64/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "3.18.17-13.el7.x86_64", arch: "amd64", family:
> "unix"
> CentOS Linux release 7.3.1611 (Core)
>
> On Tue, Jun 6, 2017 at 9:37 PM, Koji Kawamura 
> wrote:
>
> > +1 (binding)
> >
> > - ran through release helper
> > - ran clean build, test passed without issue
> > - ran simple flow using standalone/non-secure/secure/clustered NiFi
> > - confirmed few JIRAs data flow those had been addressed within 1.3.0
> >
> >
> > On Wed, Jun 7, 2017 at 2:15 AM, Pierre Villard
> >  wrote:
> > > +1 (binding)
> > >
> > > - verified signatures, checksums, license, notice and readme.
> > > - full build on OSX, Windows 10, CentOS 7 (faced some test errors while
> > > building on Windows but the failing tests have already opened JIRAs to
> > > track them)
> > > - checked UIs changes
> > > - tested some workflows for Record based processors and SiteToSite
> > > reporting tasks
> > >
> > > Release this package as nifi-1.3.0
> > >
> > >
> > > 2017-06-06 18:55 GMT+02:00 Yolanda Davis :
> > >
> > >> +1 (binding)
> > >>
> > >> -ran through release helper, verified hashes, signatures, checked
> > README,
> > >> LICENSE, NOTICE
> > >> -ran build and tested with flows using QueryRecord
> > (RecordReaders/Writers
> > >> etc), worked as expected.
> > >>
> > >> On Tue, Jun 6, 2017 at 12:44 PM, Joe Skora  wrote:
> > >>
> > >> > +1 (binding)
> > >> >
> > >> > * Verified GPG signature.
> > >> > * Verified hashes.
> > >> > * Built source with contrib-check profile.
> > >> > * Verified README, NOTICE, and LICENSE.
> > >> > * Verified commit ID.
> > >> > * Verified RC was branched from master.
> > >> > * Ran binary and tested basic functionality.
> > >> >
> > >> >
> > >> > On Tue, Jun 6, 2017 at 10:27 AM, Bryan Bende 
> > wrote:
> > >> >
> > >> > > +1 (binding) Release this package as nifi-1.3.0
> > >> > >
> > >> > > - Ran through release helper and everything checked out
> > >> > > - Ran some test flows with HDFS processors and verified they
> worked
> > >> > > correctly
> > >> > >
> > >> > > On Tue, Jun 6, 2017 at 12:42 AM, James Wing 
> > wrote:
> > >> > > > +1 (binding).  I ran through the release helper -- verified
> > signature
> > >> > and
> > >> > > > hashes, ran the full build, checked readme, license, and notice
> in
> > >> > source
> > >> > > > and output.  I ran a simple flow and verified that it still
> works.
> > >> > > >
> > >> > > > Thanks for putting this release together, Matt.
> > >> > > >
> > >> > > > James
> > >> > > >
> > >> > > >
> > >> > > > On Mon, Jun 5, 2017 at 10:54 AM, Matt Gilman <
> mcgil...@apache.org
> > >
> > >> > > wrote:
> > >> > > >
> > >> > > >> Hello,
> > >> > > >>
> > >> > > >>
> > >> > > >> I am pleased to be calling this vote for the source release of
> > >> Apache
> > >> > > NiFi
> > >> > > >> nifi-1.3.0.
> > >> > > >>
> > >> > > >>
> > >> > > >> The source zip, including signatures, digests, etc. can be
> found
> > at:
> > >> > > >>
> > >> > > >> https://repository.apache.org/content/repositories/
> > >> orgapachenifi-1108
> > >> > > >>
> > >> > > >>
> > >> > > >> The Git tag is nifi-1.3.0-RC1
> > >> > > >>
> > >> > > >> The Git commit ID is ddb73612bd1512d8b2151b81f9aa40811bca2aaa
> > >> > > >>
> > >> > > >> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> > >> > > >> ddb73612bd1512d8b2151b81f9aa40811bca2aaa
> > >> > > >>
> > >> > > >>
> > >> > > >> Checksums of nifi-1.3.0-source-release.zip:
> > >> > > >>
> > >> > > >> MD5: 8b115682ac392342b9edff3bf0658ecb
> > >> > > >>
> > >> > > >> SHA1: f11cdebbabdc0d8f1f0dd4c5b880ded39d17f234
> > >> > > >>
> > >> > > >> SHA256: 9ba5565729d98c472c31a1fdbc44e9
> > >> dc1eee87a2cf5184e8428743f75314
> > >> > > 5b7f
> > >> > > >>
> > >> > > >>
> > >> > > >> Release artifacts are signed with the following key:
> > >> > > >>
> > >> > > >> https://people.apache.org/keys/committer/mcgilman.asc
> > >> > > >>
> > >> > > >>
> > >> > > >> KEYS file available here:
> > >> > > >>
> > >> > > >> https://dist.apache.org/repos/dist/release/nifi/KEYS
> > >> > > >>
> > >> > > >>
> > >> > > >> 110 issues were closed/resolved for this release:
> > >> > > >>
> > >> > > >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > >> > > >> projectId=12316020&version=12340498
> > >> > > >>
> > >> > > >>
> > >> > > >> Release note highlights can be found here:
> > >> > > >>
> > >> > > >> https://cwiki.apache.org/confluence/display/NIFI/
> > >> > > >> Release+Notes#ReleaseNotes-Version1.3.0
> > >> > > >>
> > >> > > >>
> > 

WAL Repository - Deleted prov and TOC files held by the NiFi process

[Andre]

Folks,

Is anyone aware of conditions where the WAL repo may hold deleted files
open?


Cheers

Re: [ANNOUNCE] New Apache NiFi PMC Member - Koji Kawamura

[Andre]

Congratulations Koji

Cheers




On Sat, May 27, 2017 at 12:09 PM, Tony Kurc  wrote:

> On behalf of the Apache NiFi PMC, I am pleased to announce that Koji
> Kawamura has accepted the PMC's invitation to join the Apache NiFi PMC.  We
> greatly appreciate all of Koji's hard work and generous contributions to
> the project. We look forward to continued involvement in the project.
>
> Koji has been contributing code NiFi since 2015, taking point on important
> parts of the project like improving site-to-site. Beyond code, Koji is
> active in building the community through blogging on technical topics,
> generating interest on social media (@apachenifi #NiFi) like twitter, and
> giving talks at conferences. Koji has been keeping the project strong by
> being active in reviewing new contributions and verifying releases.
>
> Please join us in congratulating and welcoming Koji to the Apache NiFi PMC.
>
> Congratulations and welcome, Koji!
>
> -- Tony
>

Re: [ANNOUNCE] New Apache NiFi PMC Member - Pierre Villard

[Andre]

Pierre,

Your contribution to NiFi takes many forms but I would like to thank you
for the amazing blogs you have put together!

I am sure many in the user and dev lists will vouch that your blog contains
some of the first posts people will refer to when they start to use NiFi
and when they try to step up the ladder on their use of the functionality
of the platform.

Thank you and welcome aboard!!!


On Tue, May 23, 2017 at 5:17 AM, Tony Kurc  wrote:

> On behalf of the Apache NiFi PMC, I am pleased to announce that Pierre
> Villard has accepted the PMC's invitation to join the Apache NiFi PMC.  We
> greatly appreciate all of Pierre's hard work and generous contributions to
> the project. We look forward to continued involvement in the project.
>
> Pierre has been a long time code contributor to both NiFi and MiNiFi. In
> addition to that, he's spurred the community through his blogging -
> teaching people how to use, operate, and monitor NiFi. He's a prolific
> reviewer of contributions, and active in the important task of verifying
> releases.
>
> Please join us in congratulating and welcoming Pierre to the Apache NiFi
> PMC.
>
> Congratulations and welcome, Pierre!
>
> -- Tony
>

Re: [ANNOUNCE] New Apache NiFi PMC Member - Yolanda Davis

[Andre]

Yolanda,

Congratulations and it is great to have you onboard!

Cheers

On Thu, May 18, 2017 at 12:11 PM, Tony Kurc  wrote:

> On behalf of the Apache NiFi PMC, I am pleased to announce that Yolanda
> Davis has accepted the PMC's invitation to join the Apache NiFi PMC.  We
> greatly appreciate all of Yolanda's hard work and generous contributions to
> the project. We look forward to continued involvement in the project.
>
> I know many people have interacted with Yolanda maybe at one of her
> conference talks, or had the pleasure of working with her on reviews, or
> used some of the features she's contributed and put a lot of work into
> making them well tested and easy to use. If so, I think you'd agree it is
> hard to interact with Yolanda and not get excited about not only NiFi but
> the entire Apache way. If not, you're in for a treat when you get the
> chance!
>
> Please join us in congratulating and welcoming Yolanda to the Apache NiFi
> PMC.
>
> Congratulations and welcome, Yolanda!
>
> -- Tony
>

Re: Picking up and working on an issue

[Andre]

Yes it is a permissions thing. Will assign it

On Wed, May 10, 2017 at 10:50 PM, Rich Brantingham <
rich.branting...@surevine.com> wrote:

> Hi,
>
> I’d like to assign myself to an issue (https://issues.apache.org/
> jira/browse/NIFI-1207 ),
> but don't seem to be able to do it via Jira - I assume it’s a permissions
> thing?
>
> Shall I leave a comment saying that I and others are working on it, or is
> there anyone able to assign it to me?
>
> Thanks,
>
> Rich
>
>
>
>

Re: [VOTE] Release Apache NiFi 1.2.0 (RC2)

[Andre]

Bryan,

Thank you for putting this together. Tested on flow with a diverse set of
processors.

+1 binding.



Only notes:

- I believe the release notes should mention the change of the randomness
source. While I suspect most will simply ignore, I wouldn't be surprised
some people will develop some anxiety about the move from /dev/random to
/dev/urandom

- Had to rebuild twice due to test failures (provenance tests if I recall
correctly).

- I believe this is by design but on a clustered environment, the upgrade
from 1.2.0-SNAPSHOT to 1.2.0 required cluster bounce (i.e. downtime).




On Sat, May 6, 2017 at 12:07 PM, Bryan Bende  wrote:

> Hello,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi nifi-1.2.0 (RC2).
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1104
>
> The Git tag is nifi-1.2.0-RC2
> The Git commit ID is 3a605af8e0ac024fb0ba67262d49dab2727b2576
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> 3a605af8e0ac024fb0ba67262d49dab2727b2576
>
> Checksums of nifi-1.2.0-source-release.zip:
> MD5: 90e298a9e23a9dab65358daddd8b5990
> SHA1: e138941f576bdb1dff17df6674c19ffae3ef6719
> SHA256: c18398800c435dabff9f45ec55a450ca78c3c1aec222aa295c0e2057912feeb6
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/bbende.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 381 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12316020&version=12338432
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-Version1.2.0
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test.  The please vote:
>
> [ ] +1 Release this package as nifi-1.2.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because because...
>

Re: Extending 1.2.0 Records. Serialize to supported format or Extend?

[Andre]

Joe,

My question wasn't very clear but I think you still managed to provide the
insight I needed!

I will give it a go creating a reader to see if I get a better view on how
those work.

Cheers

On Sat, May 6, 2017 at 1:22 PM, Joe Witt  wrote:

> Andre
>
> If I understand the question correctly the ask is how can you extend
> the existing set of readers and writers.  This is accomplished by
> implementing controller services.  You can build them in Java or you
> can build them using the scripting language support for record
> readers/writers.
>
> You'd want a reader understands how to deserialize the bytes of a CEF
> formatted file into record objects and a writer that understands how
> to take record objects and serialize them into CEF.  Same for evtx.
> As you deserialize and parse the events you identify fields and values
> and create the record structure.
>
> Any of the processors that support the record concept have a reader
> and writer configured on them.  If you wanted to end up with JSON
> records then you'd plug-in a json writer.  You can get a sense of much
> of this from unit tests and I'd also be happy to put up a template
> that shows off how to configure a flow that reads from NiFi's own
> provenance stream, turns it into JSON, Avro, plaintext, and xml all at
> once.  I'll try to do that tomorrow.  That should make it a lot more
> clear.
>
> Thanks
>
> On Fri, May 5, 2017 at 10:51 PM, Andre  wrote:
> > All,
> >
> > I was doing some reading during my spare time and the record related
> > feature set is truly exciting.
> >
> > However I was wondering, how should one extend the existing range of
> > readers?
> >
> > I use for example ParseCEF and ParseEvtx. CEF while not particularly
> simple
> > to parse and validate using RegEx'es (and Grok) is a reasonably
> structured
> > format (with the same field supporting 9 different date formats...)
> >
> > ParseCEF currently exposes the parsed and validade CEF payload either to
> > attributes or JSON to content file. And hence my question:
> >
> > Is this the desired path of action:
> >
> > ParseCEF -> JSON content -> ConvertRecord (using JsonPathReader) ->
> whatever
> >
> > ?
> >
> > Or should extension make more sense?
> >
> >
> > Cheers
>

Extending 1.2.0 Records. Serialize to supported format or Extend?

[Andre]

All,

I was doing some reading during my spare time and the record related
feature set is truly exciting.

However I was wondering, how should one extend the existing range of
readers?

I use for example ParseCEF and ParseEvtx. CEF while not particularly simple
to parse and validate using RegEx'es (and Grok) is a reasonably structured
format (with the same field supporting 9 different date formats...)

ParseCEF currently exposes the parsed and validade CEF payload either to
attributes or JSON to content file. And hence my question:

Is this the desired path of action:

ParseCEF -> JSON content -> ConvertRecord (using JsonPathReader) -> whatever

?

Or should extension make more sense?


Cheers

Re: [VOTE] Release Apache NiFi 1.2.0

[Andre]

+1 binding. Release this package as nifi-1.2.0

On Fri, May 5, 2017 at 6:27 AM, Bryan Bende  wrote:

> Hello,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi nifi-1.2.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1103
>
> The Git tag is nifi-1.2.0-RC1
> The Git commit ID is 243ed742baddd98d8dcbb9752e07980b04ae2b89
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> 243ed742baddd98d8dcbb9752e07980b04ae2b89
>
> Checksums of nifi-1.2.0-source-release.zip:
> MD5: c6e02b78189e390656b4f6eb02fd573a
> SHA1: 1b206c96bcf9877a4119fd97cc4ea10425ada3bb
> SHA256: 8dee705d04d872112bc9143ea73f41be0f233e7bcffc913762f48f4482daadeb
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/bbende.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 375 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12316020&version=12338432
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-Version1.2.0
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.
> The please vote:
>
> [ ] +1 Release this package as nifi-1.2.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because because...
>

Re: NiFi unsecure cluster setup issue on Windows.

[Andre]

Hi,



could it be that the windows firewall is enabled and blocking the ZK
communication between nodes?

Cheers

On Wed, May 3, 2017 at 10:53 PM, shahbazatta  wrote:

> Hi,
> I am trying to setup the 3 node NiFi but i am unable to set it up. I
> already
> setup it on ubuntu and it is working fine but the same configurations not
> working on windows machines.
>
> I tried restarting all nodes, nifi service, etc but nothing works:
>
> Failed to determine which node is elected active Cluster Coordinator:
> ZooKeeper reports the address as winifi01:
>
> whereas, winifi01 node showing cluster of 1/1. i also verify the hosts file
> and zookeeper properties
>
> zookeer.properties
> server.1=winifi01:2888:3888
> server.2=winifi02:2888:3888
> server.3=winifi03:2888:3888
>
>
>
>
>
> --
> View this message in context: http://apache-nifi-developer-
> list.39713.n7.nabble.com/NiFi-unsecure-cluster-setup-issue-
> on-Windows-tp15631.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: Closing in on a NiFi 1.2.0 release?

[Andre]

All,

For some reason my canvas did not refresh after a process bounce (which
generally occurs) but reloading page allows for modifications.

Cheers

On Wed, May 3, 2017 at 7:43 AM, Andre  wrote:

> folks,
>
> I was just working to debug the final thorns found reviewing NIFI-3726 and
> noticed an odd behavior and wanted to confirm.
>
> If I recall correctly in the past users could simply replace a processor
> NAR file and even if that NAR existed the flow would continue to work.
>
> I just replaced
>
> cp ~/nifi/nifi-nar-bundles/nifi-cybersecurity-bundle/nifi-
> cybersecurity-nar/target/nifi-cybersecurity-nar-1.2.0-SNAPSHOT.nar
> ~/devel/nifi-1.2.0-SNAPSHOT/lib/nifi-cybersecurity-nar-1.2.0-SNAPSHOT.nar
>
> (note the different ~/nifi ~/devel used to ensure I don't explode the rest
> of the already compiled components).
>
> When I try to make changes to the flow I am displayed with the following
> error:
>
> [image: Inline image 1]
>
> This happens even when I try to drag and drop connected processors around
> the canvas.
>
>
> Oddly enough I can still add and delete components to the canvas but
> whatever touches the tainted processor cannot be modified at all.
>
> Examples of messages:
>
> *Attempt to move*
>
> Component Position
> [5, cb0a31ac-015b-1000-7473-873a47eb702e, 
> cb0a52ab-015b-1000-e43a-f6293a9ae99d]
> is not the most up-to-date revision. This component appears to have been
> modified
>
>
> *Attempt to delete a downstream processor*
> Error
> [1, cb0a31ac-015b-1000-7473-873a47eb702e, 
> cb0b2ae4-015b-1000-35a8-9eaf6a45fc6a]
> is not the most up-to-date revision. This component appears to have been
> modified
>
>
> I don't have a 1.1.0 instance around me at the moment but I vaguely
> remember being able to do that in the past.
>
> Can someone confirm this is new and expected behavior?
>
> Cheers
>
>
> On Wed, May 3, 2017 at 5:54 AM, Andy LoPresto 
> wrote:
>
>> I’ll review & merge as soon as they are available.
>>
>> Andy LoPresto
>> alopre...@apache.org
>> *alopresto.apa...@gmail.com *
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On May 2, 2017, at 3:51 PM, Bryan Bende  wrote:
>>
>> Thanks Drew. These seem like good candidates for the release.
>>
>> On Tue, May 2, 2017 at 3:42 PM, Andrew Lim 
>> wrote:
>>
>> There are three doc updates/additions that would be great to include in
>> the RC:
>>
>> https://issues.apache.org/jira/browse/NIFI-3701
>> https://issues.apache.org/jira/browse/NIFI-3773
>> https://issues.apache.org/jira/browse/NIFI-3774
>>
>> Sarah Olson and I have been working on these.  We should have PRs
>> submitted for them very soon.
>>
>> -Drew
>>
>>
>> On May 2, 2017, at 2:11 PM, Aldrin Piri  wrote:
>>
>> Haven't had much luck in getting our Docker efforts incorporated into
>> Docker Hub.  As a result I have created an issue to track that integration
>> [1] and resolved the original issue.
>>
>> We can evaluate our options and figure out the best path forward.  At this
>> time procedures are not yet well established within ASF to support
>> configuring these builds.
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-3772
>>
>> On Tue, May 2, 2017 at 11:13 AM, Andrew Lim 
>> wrote:
>>
>> I will be making updates to the Release Notes and Migration Guidance doc
>> regarding the TLS 1.2 version support.  Tracked by:
>>
>> https://issues.apache.org/jira/browse/NIFI-3720
>>
>>
>> -Drew
>>
>>
>> On May 2, 2017, at 11:08 AM, Joe Witt  wrote:
>>
>> Those are great updates.  I'd recommend we avoid highlighting the
>> versions of UI components though.
>>
>> Thanks
>>
>>
>> On Tue, May 2, 2017 at 11:03 AM, Scott Aslan 
>>
>> wrote:
>>
>> Hey Bryan,
>>
>> Please include the following in the release notes:
>>
>>
>> - Core UI
>>- Circular references have been removed and the code modularized.
>>- Upgraded Node version to 6.9.3.
>>- Upgraded npm version to 3.10.10.
>>- Upgraded jQuery version to 3.1.1.
>>- Upgraded D3 version to 3.5.17.
>>- Reduced download size by removing bundled dependencies.
>> - User Experience Improvements
>> - Ever wish that it was easier to align components on the canvas? Me
>>too...and now you can!
>>- We now provide deep links to any component(s) on the canvas. This
>>will help make collaborating and sharing more natural.
>

Re: Closing in on a NiFi 1.2.0 release?

[Andre]

folks,

I was just working to debug the final thorns found reviewing NIFI-3726 and
noticed an odd behavior and wanted to confirm.

If I recall correctly in the past users could simply replace a processor
NAR file and even if that NAR existed the flow would continue to work.

I just replaced

cp
~/nifi/nifi-nar-bundles/nifi-cybersecurity-bundle/nifi-cybersecurity-nar/target/nifi-cybersecurity-nar-1.2.0-SNAPSHOT.nar
~/devel/nifi-1.2.0-SNAPSHOT/lib/nifi-cybersecurity-nar-1.2.0-SNAPSHOT.nar

(note the different ~/nifi ~/devel used to ensure I don't explode the rest
of the already compiled components).

When I try to make changes to the flow I am displayed with the following
error:

[image: Inline image 1]

This happens even when I try to drag and drop connected processors around
the canvas.


Oddly enough I can still add and delete components to the canvas but
whatever touches the tainted processor cannot be modified at all.

Examples of messages:

*Attempt to move*

Component Position
[5, cb0a31ac-015b-1000-7473-873a47eb702e,
cb0a52ab-015b-1000-e43a-f6293a9ae99d] is not the most up-to-date revision.
This component appears to have been modified


*Attempt to delete a downstream processor*
Error
[1, cb0a31ac-015b-1000-7473-873a47eb702e,
cb0b2ae4-015b-1000-35a8-9eaf6a45fc6a] is not the most up-to-date revision.
This component appears to have been modified


I don't have a 1.1.0 instance around me at the moment but I vaguely
remember being able to do that in the past.

Can someone confirm this is new and expected behavior?

Cheers


On Wed, May 3, 2017 at 5:54 AM, Andy LoPresto  wrote:

> I’ll review & merge as soon as they are available.
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On May 2, 2017, at 3:51 PM, Bryan Bende  wrote:
>
> Thanks Drew. These seem like good candidates for the release.
>
> On Tue, May 2, 2017 at 3:42 PM, Andrew Lim 
> wrote:
>
> There are three doc updates/additions that would be great to include in
> the RC:
>
> https://issues.apache.org/jira/browse/NIFI-3701
> https://issues.apache.org/jira/browse/NIFI-3773
> https://issues.apache.org/jira/browse/NIFI-3774
>
> Sarah Olson and I have been working on these.  We should have PRs
> submitted for them very soon.
>
> -Drew
>
>
> On May 2, 2017, at 2:11 PM, Aldrin Piri  wrote:
>
> Haven't had much luck in getting our Docker efforts incorporated into
> Docker Hub.  As a result I have created an issue to track that integration
> [1] and resolved the original issue.
>
> We can evaluate our options and figure out the best path forward.  At this
> time procedures are not yet well established within ASF to support
> configuring these builds.
>
> [1] https://issues.apache.org/jira/browse/NIFI-3772
>
> On Tue, May 2, 2017 at 11:13 AM, Andrew Lim 
> wrote:
>
> I will be making updates to the Release Notes and Migration Guidance doc
> regarding the TLS 1.2 version support.  Tracked by:
>
> https://issues.apache.org/jira/browse/NIFI-3720
>
>
> -Drew
>
>
> On May 2, 2017, at 11:08 AM, Joe Witt  wrote:
>
> Those are great updates.  I'd recommend we avoid highlighting the
> versions of UI components though.
>
> Thanks
>
>
> On Tue, May 2, 2017 at 11:03 AM, Scott Aslan 
>
> wrote:
>
> Hey Bryan,
>
> Please include the following in the release notes:
>
>
> - Core UI
>- Circular references have been removed and the code modularized.
>- Upgraded Node version to 6.9.3.
>- Upgraded npm version to 3.10.10.
>- Upgraded jQuery version to 3.1.1.
>- Upgraded D3 version to 3.5.17.
>- Reduced download size by removing bundled dependencies.
> - User Experience Improvements
> - Ever wish that it was easier to align components on the canvas? Me
>too...and now you can!
>- We now provide deep links to any component(s) on the canvas. This
>will help make collaborating and sharing more natural.
>- Users will enjoy a better understanding of the scope of
>
> Controller
>
>Services through an improved experience.
>- All actions available on the operate palette are now also
>
> available
>
>under the context menu too!
>
> Thanks!
>
> On Tue, May 2, 2017 at 10:17 AM, Bryan Bende  wrote:
>
> All,
>
> Looks like we are closer than we have ever been!
>
> Down to three outstanding JIRAs...
>
> - NIFI-1833 (Azure Processors) - Active review and discussion
> occurring, appears to be close
> - NIFI-3260 (Official Docker Image) - Active discussion, looks like
> this may be moved from the release to a separate effort
> - NIFI-3765 (Status operation for NodeManager) - Active review, looks
> like it could be merged relatively soon
>
> Once these are finalized we should be able to start the RC process.
>
> I've started putting together the release notes on the Wiki (still a
> work in progress):
> https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-Version1.2.0
>
> If anyone knows of anything that should be highlight

[CALL FOR ACTION] Peer reviews & testing of new code

[Andre]

Dev,

Those following the git commit log probably noticed that there has been a
lot of action by some of the developers behind some significant
improvements in 1.2.0.

As these developers get busy I would like to remind the rest of the list
about importance of our contribution around code review and *testing* of
[proposed] changes to the NiFi code base.

So long history short. No matter if you have committer privilege to the
NiFi git, you can still contribute to this great community.

You can still contribute by providing extra testing of the introduced code
(I am myself running SNAPSHOT in prod-like conditions with the new
provenance WAL and it is so fast that I reckon you should be doing the
same...).


You can contribute by helping with the peer review process. [1]


So if you have some cycles available and is keen be more active in our
vibrant community, feel free to get involved!



Cheers!



[1] I admit that some of these PRs is daunting to the non-initiated (myself
included), but trust me when I say that some are perfectly within reach of
those that are familiar with NiFi and have also dealt with things like:
HDFS, Azure Cloud, Websockets, etc etc etc. (To be honest, sometimes the
contribution may be as simple as spell checking!!)

Re: [DISCUSS] Component deprecation annotation

[Andre]

All,

Thank you for the feedback.

I will extend the PR so that the required endpoints are in place but will
leave the effective UI design to our UX experts.

Hopefully I should be able to use the @Restricted as a basis for this work.
Otherwise I will shout for some guidance.

Cheers


On Mon, May 1, 2017 at 11:23 AM, Joe Witt  wrote:

> Agreed on both the icon being preferable as the vehicle to show this
> for processors on the graph instead of color and also in emphasizing
> this during processor selection.
>
> Thanks
>
> On Sun, Apr 30, 2017 at 7:21 PM, Matt Burgess  wrote:
> > Visual indication on existing processors is a good thing, but IMO we'll
> definitely need indication on the processor selection dialog, so the user
> will know that we suggest they choose an alternative.
> >
> > Regards,
> > Matt
> >
> >> On Apr 30, 2017, at 7:12 PM, Jeff  wrote:
> >>
> >> I think an icon on the processor would be best to denote an upcoming
> >> deprecation.  That leaves all colors available to the flow design, since
> >> any color may have specific meaning to any current flow out there.
> >>
> >>> On Sat, Apr 29, 2017 at 3:58 PM Juan Sequeiros 
> wrote:
> >>>
> >>> In same train of though maybe make it default to some color other than
> >>> white, instinct says red but that might be used to represent an
> important
> >>> processor. So suggest maybe default beige?
> >>>
> >>>
> >>> On Sat, Apr 29, 2017 at 8:20 AM Andy LoPresto <
> alopresto.apa...@gmail.com>
> >>> wrote:
> >>>
> >>>> I'd leave the graphics work/decisions to someone like Rob Moran, but I
> >>> see
> >>>> it as similar to the restricted components -- an icon on the
> processors
> >>> on
> >>>> the canvas and the dialog to add components, with additional
> explanation
> >>>> (and maybe a target release for full deprecation) in the help notes.
> >>>>
> >>>> Andy LoPresto
> >>>> alopre...@apache.org
> >>>> alopresto.apa...@gmail.com
> >>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >>>>
> >>>>> On Apr 29, 2017, at 05:16, Andre  wrote:
> >>>>>
> >>>>> dev,
> >>>>>
> >>>>> In light of the some recent deprecations (ListenLumberjack) and some
> >>>>> potential deprecation (PutJMS/GetJMS) I started some progress on how
> to
> >>>>> document the depreciation of NiFi components using annotations.
> >>>>>
> >>>>> In the absence of any better name I called the annotation
> >>>>> @DeprecationWarning (so not overlap with Java's @Deprecated)
> >>>>>
> >>>>> The suggested modifications are part of PR#1718 and I am keen to hear
> >>>> your
> >>>>> thoughts.
> >>>>>
> >>>>> While the documentation can be displayed as part of the usage,
> however
> >>> I
> >>>>> would imagine the frequency a user browses to the usage pages will
> >>> reduce
> >>>>> as her/him becomes familiar with the components.
> >>>>>
> >>>>> In light of this it may be a good idea to use  the web UI to
> highlight
> >>> a
> >>>>> processor has been deprecated.
> >>>>>
> >>>>> Would anyone suggest a way of doing so without disrupting the user
> >>>>> experience?
> >>>>>
> >>>>> Cheers
> >>>>
> >>>
>

[DISCUSS] Component deprecation annotation

[Andre]

dev,

In light of the some recent deprecations (ListenLumberjack) and some
potential deprecation (PutJMS/GetJMS) I started some progress on how to
document the depreciation of NiFi components using annotations.

In the absence of any better name I called the annotation
@DeprecationWarning (so not overlap with Java's @Deprecated)

The suggested modifications are part of PR#1718 and I am keen to hear your
thoughts.

While the documentation can be displayed as part of the usage, however I
would imagine the frequency a user browses to the usage pages will reduce
as her/him becomes familiar with the components.

In light of this it may be a good idea to use  the web UI to highlight a
processor has been deprecated.

Would anyone suggest a way of doing so without disrupting the user
experience?

Cheers

Re: whats new in nifi 1.2.0 / careful use of marks and being clear about non released features

[Andre]

Anshuman,

The GCS processors have been merged to master a few weeks ago so in my
personal opinion it should be a fairly safe bet to assume they will make to
1.2.0. Emphasis on assume. :-)

>From what I gather the code base is reaching stability and as per email
from Bryan earlier this week we should soon see the RC leaving the oven.

Not sure if looked at the merge history lately and I certainly don't want
to set high expectations But the changes since 1.1.2 are looking
amazing... Like... Truly amazing.

But as I said, don't want to set high expectations so hold tight as Bryan
continues to roast the RC.

Cheers



On Fri, Apr 28, 2017 at 11:01 PM, Anshuman Ghosh <
anshuman.ghosh2...@gmail.com> wrote:

> Hello Joe,
>
> I was about to write an email to the community and just then I have
> received this email. Thank you!
>
> I would like to know whether GCS processors are available or not?
> We have a requirement to use them.
> ​From where I get this latest version as the one we have downloaded doesn't
> seem to have GCS processors.​
>
> ​Thanking you in advance!​
>
> ​
> __
>
> *Kind Regards,*
> *Anshuman Ghosh*
> *Contact - +49 179 9090964*
>
>
> On Fri, Apr 28, 2017 at 2:45 PM, Joe Witt  wrote:
>
> > Team,
> >
> > Just noticed these slides
> > https://www.slideshare.net/KojiKawamura/whats-newnifi120
> >
> > While the deck itself seems quite solid and does cover a lot of really
> > important materials we must be very careful in presenting such
> > materials before the release is an officially PMC voted upon artifact.
> >
> > The note on the description is good "Summarizes new capabilities added
> > to Apache NiFi 1.2.0 (soon to be released)." but the slides themselves
> > need to adhere to the naming guidelines and need to make it very clear
> > that the features/fixes/etc.. being described are not released and
> > subject to change.
> >
> > The first use of naming should always call out the proper full name
> > which would be Apache NiFi.
> >
> > Thanks
> > Joe
> >
>

Making NiFi upgrade smoother

[Andre F de Miranda]

dev,

What do you think about the idea of allowing an user to configure overrides
for the "master" configuration files in order to smooth upgrades?

In my opinion candidates would be_ bootstrap.conf_ and  _nifi.properties_

as they contain the links to all other files (logback.xml excluded).

Upon starting bootstrap and nifi would load first the standard files as it
does today, but once the load is done, the process would proceed to load
the local overrides (reaching its final configuration).

This way that whenever a user decides to upgrade, it would simply copy or
link the override file to the NiFi folder and voilà!

I realise this is not particularly a new idea, (YARN uses the "*-site.xml"
to configure local properties and so does a number of other JAVA based
applications I have dealt with) so I am keen to hear your thoughts

Cheers

Making NiFi upgrade smoother

[Andre]

dev,

What do you think about the idea of allowing an user to configure overrides
for the "master" configuration files in order to smooth upgrades?

In my opinion candidates would be_ bootstrap.conf_ and  _nifi.properties_

as they contain the links to all other files (logback.xml excluded).

Upon starting bootstrap and nifi would load first the standard files as it
does today, but once the load is done, the process would proceed to load
the local overrides (reaching its final configuration).

This way that whenever a user decides to upgrade, it would simply copy or
link the override file to the NiFi folder and voilà!

I realise this is not particularly a new idea, (YARN uses the "*-site.xml"
to configure local properties and so does a number of other JAVA based
applications I have dealt with) so I am keen to hear your thoughts

Cheers

Re: ezmlm warning messages threatening removal from list

[Andre]

Joe,

I get them as well. I strongly suspect is caused by your server (GMAIL)
bouncing a ezlm message due to its original sender or perhaps message
volumes.

I happen to just have received one of those emails a few minutes ago
(regarding another NiFi list).

I also believe the issue tends to solve by itself when ezlm manages to send
you a message to confirm you are still there.

On Mon, Apr 24, 2017 at 8:36 PM, Joe Skora  wrote:

> Is it just me or does everyone get warning emails from the ezmlm program
> saying that messages from dev@nifi.apache.org (or user@, etc.) have been
> bouncing and that further bounces will result in removal from the list
> without notice?
>

Re: load balancer on nifi cluster

[Andre]

Pradeep,

As Bryan mentioned, load balancing of clusters boils down to the input
protocol you are using and desired load balance outcome.

In general if using TCP based load balancing you will have to make up your
mind on how to configure your load balancer. Common approaches are round
robin, byte counts, time to response or health cheacks.

Since NiFi ListenSyslog processor doesn't have any particular requirements
for load balancing (and assuming you are happy with a round robin setup)
you could simply put your TCP based load balancer (F5, HAProxy, ELB, etc)
in front of the cluster nodes as assign each new session to one of the
nodes as you would do with a TCP server.

One of our commiters Koji has put together a great example on how to
achieve load balancing with NiFi and HA proxy (adjusting it to ELB or F5s
shouldn't be very hard) and docker.

http://ijokarumawak.github.io/nifi/2016/11/01/nifi-cluster-lb/


Round robin by itself would not solve the issue of spreading the load
across nodes but if I recall correctly, you could use something like

ListenSyslog -> Remote Process Group (i.e. Input Port)

Input port -> "Do something" -> "Do more stuff" -> etc


The rationale here is since ListenSyslog is reasonably lightweight you may
still be able to complement a sub-optimal load balancing strategy, (e.g.
round robin)  with Site-to-Site cluster aware load balancing mechanisms.

Cheers


On Sat, Apr 22, 2017 at 12:50 AM, pradeepbill 
wrote:

> thanks for the article Bryan,
>
> ours is a push mechanism, data gets sent to an IP:PORT of a node in  the
> cluster, and we use listensyslog processor to listen on that port, Yes ,we
> are trying to load balance data, and not to over work a particular node in
> the cluster.From the article it says we can create a load balancer before
> the cluster, where can I get info on that ?.
>
> Thanks again.
> Pradeep
>
>
>
> --
> View this message in context: http://apache-nifi-developer-l
> ist.39713.n7.nabble.com/load-balancer-on-nifi-cluster-tp15511p15513.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: NiFi Cluster on Docker (Kubernetes) - HTTPS issues

[Andre]

Johny,

Tell me more... tell me more... have you ensure the cluster cross
communication has been set to https?

I remember seeing something like that in the summer days when I partially
setup the cluster nodes to use TLS (forgetting to do the whole job).

Can you confirm the settings for:

nifi.cluster.protocol.is.secure
nifi.remote.input.secure
nifi.web.https.port

Can you also confirm you are using wildcard certificates (or alternate
subject names) and the following are set to the correct hostnames?

nifi.web.https.host
nifi.remote.input.host

Also, can you confirm the cluster is effectively up and running? Do you see
mentions to heartbeat being made in nifi-app.log?

Cheers


On 20 Apr 2017 23:20, "Johny Travolta"  wrote:

> Hey guys,
>
> Thanks for a great product. However , to set NiFi in fully automatic way is
> a bit tricky.
> For sure the tricky part is authentication to NiFi Cluster itself (why You
> guys forced 1st user authentication with certificate? That's a huge issue
> here :) )
>
> Basically, I have created a Docker image , and I can deploy NiFi Cluster in
> automated way.
> However, we need authentication , so we must use HTTPS.
> Now, I am thinking that the problem is that all my instances are created
> the same way (from same docker image ), with same Root certificate and same
> keys...
>
> My user can login succesfully to NiFi via HTTPS, however what I see after
> is the error message below. I am not able to do anything after this:
>
>
> I know that my certificate is good, because I can login (If I will go to
> /login page) this message is visible:
>
>
> And the logs says :
>
>  at
> org.apache.nifi.cluster.coordination.http.replication.Thread
> PoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestR
> eplicator.java:802)
> ~[nifi-framework-clust er-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_121] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_121] at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1142)
> [na:1.8.0_121] at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:617)
> [na:1.8.0_121] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> Caused by: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext
> connection? at
> sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710)
> ~[na:1.8.0_121] at sun.security.ssl.InputRecord.read(InputRecord.java:527)
> ~[na:1.8.0_121] at
> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
> ~[na:1.8.0_121] at
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSo
> cketImpl.java:1375)
> ~[na:1.8.0_121] at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
> ~[na:1.8.0_121] at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
> ~[na:1.8.0_121] at
> sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
> ~[na:1.8.0_121] at
> sun.net.www.protocol.https.AbstractDelegateHttpsURLConnectio
> n.connect(AbstractDelegateHttpsURLConnection.java:185)
> ~[na:1.8.0_121] at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(
> HttpURLConnection.java:1546)
> ~[na:1.8.0_121] at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(H
> ttpURLConnection.java:1474)
> ~[na:1.8.0_121] at
> java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
> ~[na:1.8.0_121] at
> sun.net.www.protocol.https.HttpsURLConnectionImpl.getRespons
> eCode(HttpsURLConnectionImpl.java:338)
> ~[na:1.8.0_121] at
> com.sun.jersey.client.urlconnection.URLConnectionClientHandl
> er._invoke(URLConnectionClientHandler.java:253)
> ~[jersey-client-1.19.jar:1.19] at
> com.sun.jersey.client.urlconnection.URLConnectionClientHandl
> er.handle(URLConnectionClientHandler.java:153)
> ~[jersey-client-1.19.jar:1.19] ... 12 common frames omitted
>
> Thanks if You can give me a right direction to fix this!
>

Master build seems to be broken?

[Andre]

dev,

Not sure if anyone noticed but it seems like travis master has been broken
since 68c592ea43d30754ec07c42cf10563fe9db185ae ?

The broken test seems to be related to nifi-record-serialization-services

Cheers

Re: Stalled PR clean-up

[Andre]

done.

only thing I noted while double checking was that PR#959 should read PR#969
(description and remained the same).

Cheers


On Tue, Apr 11, 2017 at 2:03 AM, Joe Witt  wrote:

> Agree with James.  Go for it.  I should have done it already but would
> appreciate the assist.
>
> Thanks
>
> On Apr 10, 2017 11:51 AM, "James Wing"  wrote:
>
> > +1, please close.  It's worth adding that all of the above PRs were asked
> > to provide proof-of-life back in February.
> >
> > On Mon, Apr 10, 2017 at 8:25 AM, Andre  wrote:
> >
> > > dev,
> > >
> > > The PR queue is back at 80 contributions and unless someone suggest
> > > otherwise I would love to be able to close the following four PRs:
> > >
> > > PR#254 - NIFI-1583 Added ConvertJSONtoCSV processor (No update since
> Oct
> > > 2016)
> > > PR#293 - NIFI-1613 Initial version, try to improve conversion for
> > different
> > > SQ... (Stalled Mar 2016)
> > > PR#626 - NIFI-1922 added support for HDInsight wasb file storage to the
> > > nifi-h… (won't as there's an alternative).
> > > PR#959 - NIFI-2629 - Hide add property "+" in Processors, Controller
> > > Services and Reporting Tasks that don't allow dynamic properties  (No
> > > update since Sep 2016)
> > >
> > >
> > > Looking forward for your feedback.
> > >
> >
>

Stalled PR clean-up

[Andre]

dev,

The PR queue is back at 80 contributions and unless someone suggest
otherwise I would love to be able to close the following four PRs:

PR#254 - NIFI-1583 Added ConvertJSONtoCSV processor (No update since Oct
2016)
PR#293 - NIFI-1613 Initial version, try to improve conversion for different
SQ... (Stalled Mar 2016)
PR#626 - NIFI-1922 added support for HDInsight wasb file storage to the
nifi-h… (won't as there's an alternative).
PR#959 - NIFI-2629 - Hide add property "+" in Processors, Controller
Services and Reporting Tasks that don't allow dynamic properties  (No
update since Sep 2016)


Looking forward for your feedback.

Re: [DRAFT REPORT] Apache NiFi - April 2017

[Andre]

Joe,

Thanks for doing this.

+1



On Tue, Apr 11, 2017 at 12:28 AM, Aldrin Piri  wrote:

> +1 with the changes noted, thanks for constructing
>
> On Mon, Apr 10, 2017 at 4:25 PM, Joe Witt  wrote:
>
> > Tony
> >
> > Good call yes.
> >
> > Off-list it was also flagged to fix 'Ben' to 'Bin' and use less wordy
> > form of "appears rather strong" to "appears strong".  Making those
> > adjustments now.
> >
> > Thanks
> > Joe
> >
> > On Mon, Apr 10, 2017 at 10:21 AM, Tony Kurc  wrote:
> > > Joe,
> > > Wasn't the 1.2.0 of the nifi-nar-maven-plugin last month? Should we
> > report
> > > that if so?
> > >
> > > Tony
> > >
> > > On Mon, Apr 10, 2017 at 9:34 AM, Joe Witt  wrote:
> > >
> > >> Team,
> > >>
> > >> Please review this draft report I plan to submit to the board very
> > >> soon.  If you have feedback, things to add, etc.. please let me know.
> > >>
> > >> Thanks all!
> > >>
> > >> 
> > >>
> > >> ## Description:
> > >>  - Apache NiFi is an easy to use, powerful, and reliable system to
> > >> process and distribute data.
> > >>  - Apache MiNiFi, a child project of Apache NiFi, is an edge data
> > >> collection agent built to seamlessly integrate with and leverage the
> > >> command and control of NiFi. There are both Java and C++
> > >> implementations.
> > >> - Apache NiFi Registry, a child project of Apache NiFi, is a
> centralized
> > >> registry for key configuration items including flow versions, assets,
> > >> and extensions for Apache NiFi and Apache MiNiFi.
> > >>
> > >> ## Issues:
> > >>  - There are no issues requiring board attention at this time.
> > >>
> > >> ## Activity:
> > >>  - Apache NiFi 1.1.2 was released on February 24, 2017.
> > >>  - Apache NiFi 0.7.2 was released on February 24, 2017.
> > >>  - We have launched a security@nifi alias and security procedure
> > webpage.
> > >>  Have reported several CVEs and conducted releases to address the
> > findings.
> > >>  - We have launched a new subproject called Apache NiFi Registry.
> > >>
> > >> ## Health report:
> > >>  - The state of the community appears rather strong.  We're seeing
> > >> consistently
> > >>  high activity on mailing lists, contributions of code and review
> > >> feedback, and
> > >>  are trending again toward a very significant release.  We're seeing
> > signs
> > >> of
> > >>  maturity for the project as well with a formalized security reporting
> > and
> > >>  handling process.
> > >>  - We are closing in on a significant release of Apache NiFi and both
> > the
> > >>  Apache MiNiFi CPP and Java implementations.
> > >>
> > >> ## PMC changes:
> > >>
> > >>  - Currently 19 PMC members.
> > >>  - New PMC members:
> > >> - James Wing was added to the PMC on Mon Feb 20 2017
> > >> - Oleg Zhurakousky was added to the PMC on Mon Apr 03 2017
> > >>
> > >> ## Committer base changes:
> > >>
> > >>  - Currently 32 committers.
> > >>  - New commmitters:
> > >> - Ben Qiu was added as a committer on Tue Apr 04 2017
> > >> - Jeff Storck was added as a committer on Sat Feb 18 2017
> > >>
> > >> ## Releases:
> > >>
> > >>  - nifi-0.7.2 was released on Fri Feb 24 2017
> > >>  - nifi-1.1.2 was released on Fri Feb 24 2017
> > >>
> > >> ## Mailing list activity:
> > >>
> > >>  - We continue to see steady growth in the number of participants on
> our
> > >>  developer and user mailing lists as well as increased collaboration
> > >> between
> > >>  new community members.  We're also seeing strong interactions on
> other
> > >> sites
> > >>  such as Twitter and StackOverflow.
> > >>
> > >>  - us...@nifi.apache.org:
> > >> - 473 subscribers (up 40 in the last 3 months):
> > >> - 972 emails sent to list (790 in previous quarter)
> > >>
> > >>  - dev@nifi.apache.org:
> > >> - 343 subscribers (up 33 in the last 3 months):
> > >> - 1027 emails sent to list (855 in previous quarter)
> > >>
> > >>  - iss...@nifi.apache.org:
> > >> - 38 subscribers (up 6 in the last 3 months):
> > >> - 7271 emails sent to list (6683 in previous quarter)
> > >>
> > >>
> > >> ## JIRA activity:
> > >>
> > >>  - 452 JIRA tickets created in the last 3 months
> > >>  - 321 JIRA tickets closed/resolved in the last 3 months
> > >>
> >
>

Re: [ANNOUNCE] New Apache NiFi Committer Bin Qiu

[Andre]

congrats Bin!

Looking forward for building a NiFi-CPP cluster in the future (Late
April fool... happy to keep it as MiNiFi-CPP :-) )

Cheers



On Wed, Apr 5, 2017 at 6:17 AM, Tony Kurc  wrote:

> Apache NiFi community,
>
>
> On behalf of the Apache NiFI PMC, I am very pleased to announce that Bin
> Qiu has accepted the PMC's invitation to become a committer on the Apache
> NiFi project. We greatly appreciate all of Bin’s hard work and generous
> contributions and look forward to continued involvement in the project.
>
> Bin has been one of the driving forces behind MiNiFi - C++, contributing a
> massive amount in terms of code and helping with release verification and
> always being responsive to the community.
>
>
> Congrats, Bin!
>
>
> - Tony
>

Re: [ANNOUNCE] New Apache NiFi PMC Member - Oleg Zhurakousky

[Andre]

Oleg,

Thank you for accepting the invitation to join the PMC and welcome aboard!

Cheers

On Wed, Apr 5, 2017 at 5:51 AM, Tony Kurc  wrote:

> Apache NiFi Community,
>
> On behalf of the Apache NiFi PMC, I am pleased to announce that Oleg
> Zhurakousky has accepted the PMC's invitation to join the Apache NiFi PMC.
> We greatly appreciate all of Oleg’s hard work and generous contributions to
> the project. We look forward to his continued involvement in the project.
>
> Oleg started out contributing in late 2015, diving in deep on framework
> issues and contributing exciting new capabilities. I had the pleasure of
> reviewing some of those early contributions and always found the dialog
> with Oleg on reviews and ideas to be fun and stimulating. Since accepting
> the PMC’s invitation to become a committer, he’s maintained a steady hand
> in reviewing new contributions, staying active contributing code and on the
> mailing lists, and becoming an ever more valuable member of our community.
>
>
> Please join us in congratulating and welcoming Oleg to the Apache NiFi PMC.
>
> Congratulations and welcome, Oleg!
>

Re: Deconstructing SMTP /Mime emails

[Andre]

David,

The original email is part of the content produced by ListenSMTP itself.
The challenge with emails is that sometimes the body itself may be encoded
and this may or may not be extracted via ExtractEmailAttachments.

Do you have a particular example of message you want to process?

Cheers

On Sat, Apr 1, 2017 at 6:56 AM, DAVID SMITH 
wrote:

> Hi
> I have a scenario where I use listenSMTP to listen for emails, and then I
> use ExtractEmailHeaders and ExtractEmailAttachments to deconstruct these
> emails into their constituent parts, however, I don't seem to be able to
> get the body text of the email as either an attribute or a FlowFile. Can
> anyone help , is there a way of doing this with a standard processor or do
> I need to write a bespoke processor?
> Many thanksDave
>

Re: Processor missing when attempting Add Processor

[Andre]

Russell,

Not sure if this is a typo but have you noticed this?

p//rocessor.Processor

Cheers


On 29 Mar 2017 8:57 AM, "Russell Bateman"  wrote:

I've built a NAR containing a custom processor that loads in NiFi, but the
processor cannot be found. At the top of /TikaProcessor.java/, I have,
among other annotations, this:

@Tags( { "tika" } )

I've tried a number of things to solve this, like making it the only NAR
besides just the set NiFi 1.1.1 ships with, and wiping out all flows down
to a blank canvas. I tried removing SNAPSHOT from its version. *I see this
in **/logs/nifi-app.log/*: This is the only reference to it in any log, but
it does make me think that it's loaded:

2017-03-28 15:24:20,226 INFO [main] org.apache.nifi.nar.NarClassLoaders
Loaded NAR file: /home/russ/dev/nifi/nifi-1.1.1
/./work/nar/extensions/nifi-tika-1.0.1.nar-unpacked as class loader
org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/
nifi-tika-1.0.1.nar-unpacked]

In /resources/META_INF/services/, /org.apache.nifi.processor.Processor/
contains:

/com.imatsolutions.nifi.processor.TikaProcessor/

 I have a separate, very large project with many custom processors all of
which load fine (and I've used this one before. In fact, the only thing
I've done is remove this long-working processor from a larger set.) This
project is very small:

   nifi-tika
   +-- nar
   | +-- /nifi-tika-1.0.1.nar/
   |   `-- pom.xml (packaging is "nar")
   +-- pom.xml (packaging is "pom")
   `-- tika
+-- pom.xml (packaging is "jar")
+-- src (test, resources, etc. including
   /resources/META_INF/services/org.apache.nifi.p//rocessor.Processor/)
`-- target
`-- /tika-1.0.1.jar/

I'm not certain what else to try. My /pom.xml/ files produce a NAR. NiFi
appears to dignify it as a NAR. I keep retracing all the steps, but cannot
figure out what I've missed.

Re: maybe a handy script for working with PRs

[Andre]

Trying to save key strokes?

Just use gitg No wasted keystrokes, just use the
mouse/trackball/touchpad/whatever

:-D

On Mon, Mar 27, 2017 at 12:52 PM, Matt Burgess  wrote:

> +1 for Yetus, also this for saving keystrokes :)
>
> https://dev.to/sendra/git-faster-with-mingit
>
>
>
> > On Mar 24, 2017, at 3:58 PM, Sean Busbey  wrote:
> >
> > May I humbly suggest y'all check out the Apache Yetus project?
> >
> > http://yetus.apache.org/
> >
> > It includes a framework for running some community decided automated
> > tests against new contributions and works for both JIRA attachments
> > and PRs in github.
> >
> > http://yetus.apache.org/documentation/0.4.0/precommit-basic/
> >
> > It also has a stripped down tool for just doing the fetch / apply
> > dance called 'smart-apply-patch'.
> >
> >
> >
> >> On Thu, Mar 23, 2017 at 9:01 PM, Otto Fowler 
> wrote:
> >> Nice,
> >> I like this one because I can just dump a pr in ~/tmp or wherever -or-
> in
> >> my current.
> >>
> >> I’ll star this though :)
> >>
> >>
> >> On March 23, 2017 at 18:55:52, Andy LoPresto (alopre...@apache.org)
> wrote:
> >>
> >> Thanks Otto. I have a similar bash function I use:
> >>
> >> function gpr() {
> >> echo "Fetching and checking out new git branch pr$@..."
> >> git fetch --all
> >> git checkout master
> >> git pull upstream master
> >> git checkout "upstream/pr/$@"
> >> git checkout -b "pr$@“
> >> }
> >>
> >> Usage: For PR 1234
> >>
> >> $ gpr 1234
> >>
> >> Andy LoPresto
> >> alopre...@apache.org
> >> *alopresto.apa...@gmail.com *
> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >>
> >> On Mar 23, 2017, at 7:00 AM, Otto Fowler 
> wrote:
> >>
> >> On the metron project, many of our committers use a couple of scripts
> when
> >> working with PRs, either cloning and testing or preparing commits.
> >> I have created a version of the checkout-pr script for the nifi project,
> >> and though maybe someone would be interested.
> >>
> >> This work is a fork of Nick Allen’s original repo for Metron.
> >>
> >> https://github.com/ottobackwards/commit-pr-stuff
> >>
> >> If you look in the nifi folder, there is a checkout-nifi-pr script,
> which
> >> when called will checkout a pr for you to work with.
> >> I am not sure if you have something similar already, but I thought I
> would
> >> throw this over the wall just the same.
> >>
> >> O
>

Re: Commit d90cf84 seems to break build?

[Andre]

Aldrin,

While we may not get much granularity we can certainly hack our way by "rm
-rf ~/.m2/repository/whatever_we_want_to_delete" prior to calling maven ?

I appreciate caching is sometime a pain, but given our build currently
takes almost 40 minutes and travis-ci.org jobs will timeout at 50 minutes,
I suspect the time saved by not having to download dependencies comes very
handy.

I would suggest that we keep caching on (the thing that happened with
jB(C|c)rypt is not that usual after all...) but remove what we are testing
(i.e. ~/.m2/repository/org/nifi ) prior to build, what do you think?

Cheers


On Sun, Mar 26, 2017 at 3:38 AM, Aldrin Piri  wrote:

> Awesome, thanks for fixing it up, Bryan.
>
> I don't think we can get that kind of granularity with Travis,
> unfortunately.  However, the last time was because an artifact changed its
> name (or more specifically, casing).
>
> Not sure removing caching is the best option, but seems like the the
> optimization may not provide as much value in build speedup as the
> consistent, cleanroom environment for builds.  Just a thought.
>
> On Sat, Mar 25, 2017 at 11:18 AM, Bryan Bende  wrote:
>
> > Thanks Aldrin, I pushed the commit.
> >
> > As far as travis, I am not familiar with how it works, but can you
> > specify what to cache?
> >
> > In this case we didn't need a completely clean .m2, we just needed a
> > clean .m2/org/apache/nifi.
> >
> > On Sat, Mar 25, 2017 at 11:10 AM, Aldrin Piri 
> > wrote:
> > > Please just push to correct. Simple fixes are fine in my book.
> > >
> > > Does it make sense to potentially scrap caching in Travis?
> > >
> > > This is another time we have missed something like this that no caching
> > > would have prevented. Additionally, given the large footprint of the
> > > repository download it seems as though its benefit may be marginal as
> per
> > > https://docs.travis-ci.com/user/caching/#How-does-caching-work%3F
> > > On Sat, Mar 25, 2017 at 11:04 Bryan Bende  wrote:
> > >
> > >> Andre,
> > >>
> > >> Thanks for bringing this up.
> > >>
> > >> The standard prioritizers moved from the standard bundle to the
> > >> framework bundle, and sure enough the parent was still set as standard
> > >> bundle. We had never built with a clean repo and were getting lucky,
> > >> so I am glad you found this.
> > >>
> > >> In
> > >> nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-standard-
> > prioritizers/pom.xml
> > >>
> > >> The parent should be nifi-framework and not
> > >> nifi-standard-bundle.
> > >>
> > >> If no one objects I can push up this change, not sure if we need a
> > >> formal PR for a one line change for something thats already broken??
> > >>
> > >> Thanks,
> > >>
> > >> Bryan
> > >>
> > >> On Sat, Mar 25, 2017 at 9:33 AM, Andre  wrote:
> > >> > dev,
> > >> >
> > >> > Is anyone else having issues building master from "clean" (i.e. rm
> -rf
> > >> > ~/.m2/repositories/org/apache/nifi) after commit  d90cf84 ?
> > >> >
> > >> > My attempts to build currently yield:
> > >> >
> > >> > $ mvn -T2.0C -DskipTests=true -Pdir-only clean install
> > >> > [INFO] Scanning for projects...
> > >> > [ERROR] [ERROR] Some problems were encountered while processing the
> > POMs:
> > >> > [WARNING] 'parent.relativePath' of POM
> > >> > org.apache.nifi:nifi-standard-prioritizers:[unknown-version]
> > >> >
> > >> (/home/afucs/nifi/nifi-nar-bundles/nifi-framework-bundle/
> > nifi-framework/nifi-standard-prioritizers/pom.xml)
> > >> > points at org.apache.nifi:nifi-framework instead of
> > >> > org.apache.nifi:nifi-standard-bundle, please verify your project
> > >> structure
> > >> > @ line 17, column 13
> > >> > [FATAL] Non-resolvable parent POM for
> > >> > org.apache.nifi:nifi-standard-prioritizers:[unknown-version]: Could
> > not
> > >> > find artifact org.apache.nifi:nifi-standard-
> bundle:pom:1.2.0-SNAPSHOT
> > and
> > >> > 'parent.relativePath' points at wrong local POM @ line 17, column 13
> > >> >  @
> > >> > [ERROR] The build could not read 1 project -> [Help 1]
> > >> > [ERROR]
> > >> > [ERROR]   Th

Commit d90cf84 seems to break build?

[Andre]

dev,

Is anyone else having issues building master from "clean" (i.e. rm -rf
~/.m2/repositories/org/apache/nifi) after commit  d90cf84 ?

My attempts to build currently yield:

$ mvn -T2.0C -DskipTests=true -Pdir-only clean install
[INFO] Scanning for projects...
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[WARNING] 'parent.relativePath' of POM
org.apache.nifi:nifi-standard-prioritizers:[unknown-version]
(/home/afucs/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-standard-prioritizers/pom.xml)
points at org.apache.nifi:nifi-framework instead of
org.apache.nifi:nifi-standard-bundle, please verify your project structure
@ line 17, column 13
[FATAL] Non-resolvable parent POM for
org.apache.nifi:nifi-standard-prioritizers:[unknown-version]: Could not
find artifact org.apache.nifi:nifi-standard-bundle:pom:1.2.0-SNAPSHOT and
'parent.relativePath' points at wrong local POM @ line 17, column 13
 @
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR]   The project
org.apache.nifi:nifi-standard-prioritizers:[unknown-version]
(/home/afucs/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-standard-prioritizers/pom.xml)
has 1 error
[ERROR] Non-resolvable parent POM for
org.apache.nifi:nifi-standard-prioritizers:[unknown-version]: Could not
find artifact org.apache.nifi:nifi-standard-bundle:pom:1.2.0-SNAPSHOT and
'parent.relativePath' points at wrong local POM @ line 17, column 13 ->
[Help 2]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions,
please read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2]
http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException


However, when I interactively rebase the branch and dro pthe above
mentioned commit, build starts working again.


Cheers

[SUGGESTION] Annotation for memory intensive processors (i.e. those that load content into memory)

[Andre]

dev,

While I know we all would prefer if processors only used data streaming
within onTrigger calls, sometimes processors must load data into memory
prior to action.

Should we have an annotation to flag those unfortunate processors that for
a particular reason must load flowfile contents into memory before doing
their magic?

Cheers

[HEADS-UP] Travis is green again!

[Andre]

dev,

Just as a friendly heads up:

The last remaining issue of tests failing when using certain locales was
 fixed today when NIFI-3466 was merged into master.

This means the test units are now working with 4 different Locales without
any know issues and travis is back to green (yay!).

Moving forward it may be a good practice to ensure any new code is capable
of passing at least the current set of locales.


Cheers

Re: Kafkaesque Output port

[Andre F de Miranda]

Aldrin,



Another point for consideration is the scope of this information.  Core
NiFi flows and those components are very much about data whereas those IPs
may not necessarily be data for consumption, per se, but context that
governs how the data flow is operating.  In this case, there is a different
plane of information exchange and may deserve a discrete logical channel to
make this available.



Not sure what you mean by governing the context but maybe I should explain
myself a bit better: from a NiFi point of view the data (IPs)  are still
data to be consumed, the only difference in this case is that the data is
very small and the final delivery mechanism is a Unix command (instead of a
PutElasticSearch processor for example)

The fact we update firewall ACL from the sensors themselves is relatively
irrelevant to the overall flow.

As an example: The IPs could be sent  from sensors m1 and m2 to core and
then to a completely independent set of sensors mm1-mmN where the same data
is simply saved into disk (instead of running ExecuteScript).

I may be wrong but I was under the impression that from a minifi point of
view, what we do with the data at the end of the data flow is not a matter
of relevance, if so, then at least in this case, this would still count as
"data in movement scenario" that we try to cover.


Would you agree?

Re: Kafkaesque Output port

[Andre]

Simon,

Thank you for your comments... I was aware of the use of Kafka and
alternatives but I think that limitations aside (e. g. dificulty of
transfering files over kafka, broken provenance chain, etc) many would
refrain from using yet another piece of infra to reach the 1-n clients.

This is specially undesirable when you think that many minifi users are
likely to be using S2S to send data into NiFi

Cheers

On 17 Mar 2017 12:29 AM, "Simon Lucy"  wrote:

There are two different classes of queues really and you can't mix them
semantically.

The pubsub model

 * where ordering isn't guaranteed,
 * messages may appear at least once but can be duplicates
 * messages need to be explicitly deleted or aged
 * messages may or may not be persisted

The event model

 * ordering is necessary and guaranteed
 * messages appear only once
 * once read a message is discarded from the queue
 * messages are probably persisted

Kafka can be used for both models but Nifi Output Ports are meant to be
Event Queues. What you could do though is have an external message broker
connect to the Output Port and distribute that to subscribers. It could be
Kafka, Rabbit, AMQ AWS's SQS/SNS whatever makes sense in the context.

There's no need to modify or extend Nifi then.

S



Aldrin Piri <mailto:aldrinp...@gmail.com>
> Thursday, March 16, 2017 1:07 PMvia Postbox <https://www.postbox-inc.com/?
> utm_source=email&utm_medium=sumlink&utm_campaign=reach>
>
> Hey Andre,
>
> Interesting scenario and certainly can understand the need for such
> functionality. As a bit of background, my mind traditionally goes to
> custom controller services used for referencing datasets typically served
> up via some service. This means we don't get the Site to Site goodness and
> may be duplicating effort in terms of transiting information. Aspects of
> this need are emerging in some of the initial C2 efforts where we have this
> 1-N dispersion of flows/updates to instances, initial approaches are
> certainly reminiscent of the above. I am an advocate for Site to Site
> being "the way" data transport is done amongst NiFi/MiNiFi instances and
> this is interlaced, in part, with the JIRA to make this an extension point
> [1]. Perhaps, more simply and to frame in the sense of messaging, we have
> no way of providing topic semantics between instances and only support
> queueing whether that is push/pull. This new component or mode would be
> very compelling in conjunction with the introduction of new protocols each
> with their own operational guarantees/caveats.
>
> Some of the first thoughts/questions that come to mind are:
> * what buffering/age off looks like in context of a connection. In part,
> I think we have the framework functionality already in place, but requires
> a slightly different though process and context.
> * management of progress through the "queue", for lack of a better word,
> on a given topic and how/where that gets stored/managed. this would be the
> analog of offsets
> * is prioritization still a possibility? at first blush, it seems like
> this would no longer make sense and/or be applicable
> * what impact does this have on provenance? seems like it would still map
> correctly; just many child send events for a given piece of data
> * what would the sequence of receiving input port look like? just use run
> schedule we have currently? Presumably this would be used for updates, so
> I schedule it to check every N minutes and get all the updates since then?
> (this could potentially be mitigated with backpressure/expiration
> configuration on the associated connection).
>
> I agree there is a certain need to fulfill that seems applicable to a
> number of situations and finding a way to support this general data
> exchange pattern in a framework level would be most excellent. Look
> forward to discussing and exploring a bit more.
>
> --aldrin
>
> [1] https://issues.apache.org/jira/browse/NIFI-1820
>
>
> Andre <mailto:andre-li...@fucs.org>
> Thursday, March 16, 2017 12:27 PMvia Postbox <
> https://www.postbox-inc.com/?utm_source=email&utm_medium=su
> mlink&utm_campaign=reach>
>
> dev,
>
> I recently created a demo environment where two remote MiNiFi instances (m1
> and m2) were sending diverse range of security telemetry (suspicious email
> attachments, syslog streams, individual session honeypot logs, merged
> honeypot session logs, etc) from edge to DC via S2S Input ports
>
> Once some of this data was processed at the hub I then used Output ports to
> send contents back to the spokes, where the minifi instances use the
> flowfiles contents as arguments of OS commands (called via Gooovy
> String.execute().text via ExecuteScript).
>
> The idea

Kafkaesque Output port

[Andre]

dev,

I recently created a demo environment where two remote MiNiFi instances (m1
and m2) were sending diverse range of security telemetry (suspicious email
attachments, syslog streams, individual session honeypot logs, merged
honeypot session logs, etc) from edge to DC via S2S Input ports

Once some of this data was processed at the hub I then used Output ports to
send contents back to the spokes, where the minifi instances use the
flowfiles contents as arguments of OS commands (called via Gooovy
String.execute().text via ExecuteScript).

The idea being to show how NiFi can be used in basic security orchestration
(in this case updating m1's firewall tables with malicious IPs observed in
m2 and vice versa).


While crafting the demo I noticed the Output ports operate like queues,
therefore if one client consumed data from the port, the other was unable
to obtain the same flowfiles.

This is obviously not an issue when using 2 minifi clients (where I can
just create another output port and clone to content) but wouldn't flow
very well with hundred of clients.

I wonder if anyone would have a suggestion of how to achieve a N to 1
Output port like that? And if not, I wonder if we should create one?

Cheers

Re: Nifi Output port Use case

[Andre]

Aditya,

MapR Stream is Kafka compatible so you could in theory just push from NiFi
into the MapR cluster and get Spark to consume from there. However, if you
want to bypass MapR streams then you can use Output ports as documented
here:

https://blogs.apache.org/nifi/entry/stream_processing_nifi_and_spark

and

https://community.hortonworks.com/articles/12708/nifi-feeding-data-to-spark-streaming.html

If I recall correctly, when using direct Spark to NiFi communications, the
Spark job will behave like a NiFi site 2 site peer (be mindful of access
controls) and consume data from there.

If what you want is to upload data from the Spark job into NiFi then you
once again have a few options one of them (easiest one?) is configuring
NiFi to read straight out of MapR-FS and send data into HDFS.

Cheers



On Tue, Mar 7, 2017 at 9:32 PM, Aditya Gaurav (agaurav) 
wrote:

> Hi,
>
> We are building a system to consume an AVRO file for ingest into HDFS. We
> are planning to use MAPR-STREAM for the same and found Nifi interesting for
> handling input requests.
>
> I wanted to check if we can put a file to a Nifi port which can be read
> via a Spark job?
>
> Pls let me know.
>
>
> Thanks
> Aditya
>

Re: Is anyone else having trouble downloading jBcrypt dependency?

[Andre]

Pere,

Glad to see I am not the only one. I merged your patch

Cheers


On Sun, Mar 5, 2017 at 12:29 AM, Pere Urbón Bayes 
wrote:

> HI,
>   I also had the same problem this morning, and I pushed a path for this
> this morning, https://issues.apache.org/jira/browse/NIFI-3554, just my
> first contribution, not sure if I missed something in the process :-), let
> me know.
>
> - purbon
>
> On Sat, Mar 4, 2017 at 2:11 PM Andre  wrote:
>
> > dev,
> >
> > Seems the name has always been jBCrypt. odd
> >
> > | Maven setup
> > |
> > | Use it in your project by adding the following to your project pom.xml:
> > |
> > |
> > |de.svenkubiak
> > |jBCrypt
> > |0.4.1
> > |
> >
> > [Source: https://github.com/svenkubiak/jBCrypt#maven-setup]
> >
> >
> > Since it is quite late here in Oz and I haven't opened a JIRA ticket ( I
> > didn't want to open one in case it may be related to my environment), ff
> > confirmed by others, would anyone mind opening a JIRA ticket and merging
> > the following PR#1563 (assuming it fixes it)?
> >
> > Truly appreciated
> >
> > Cheers
> >
> >
> > On Sat, Mar 4, 2017 at 11:49 PM, Andre  wrote:
> >
> > > dev,
> > >
> > > I was wondering if anyone else is having trouble building NiFi after
> > > purging their maven repositories?
> > >
> > > $ git rebase apache/master
> > > Current branch NIFI-871 is up to date.
> > >
> > > $ mvn -U  -DskipTests=true  clean install
> > > ...
> > > [WARNING] The POM for de.svenkubiak:jBcrypt:jar:0.4.1 is missing, no
> > > dependency information available
> > > ...
> > > [ERROR] Failed to execute goal on project nifi-standard-processors:
> Could
> > > not resolve dependencies for project
> > org.apache.nifi:nifi-standard-processors:jar:1.2.0-SNAPSHOT:
> > > Could not find artifact de.svenkubiak:jBcrypt:jar:0.4.1 in central (
> > > https://repo1.maven.org/maven2) -> [Help 1]
> > >
> > > From what I gather the dependency has been renamed into jBCrypt?
> > >
> > > Cheers
> > >
> >
>

Re: Is anyone else having trouble downloading jBcrypt dependency?

[Andre]

dev,

Seems the name has always been jBCrypt. odd

| Maven setup
|
| Use it in your project by adding the following to your project pom.xml:
|
|
|de.svenkubiak
|jBCrypt
|0.4.1
|

[Source: https://github.com/svenkubiak/jBCrypt#maven-setup]


Since it is quite late here in Oz and I haven't opened a JIRA ticket ( I
didn't want to open one in case it may be related to my environment), ff
confirmed by others, would anyone mind opening a JIRA ticket and merging
the following PR#1563 (assuming it fixes it)?

Truly appreciated

Cheers


On Sat, Mar 4, 2017 at 11:49 PM, Andre  wrote:

> dev,
>
> I was wondering if anyone else is having trouble building NiFi after
> purging their maven repositories?
>
> $ git rebase apache/master
> Current branch NIFI-871 is up to date.
>
> $ mvn -U  -DskipTests=true  clean install
> ...
> [WARNING] The POM for de.svenkubiak:jBcrypt:jar:0.4.1 is missing, no
> dependency information available
> ...
> [ERROR] Failed to execute goal on project nifi-standard-processors: Could
> not resolve dependencies for project 
> org.apache.nifi:nifi-standard-processors:jar:1.2.0-SNAPSHOT:
> Could not find artifact de.svenkubiak:jBcrypt:jar:0.4.1 in central (
> https://repo1.maven.org/maven2) -> [Help 1]
>
> From what I gather the dependency has been renamed into jBCrypt?
>
> Cheers
>

Is anyone else having trouble downloading jBcrypt dependency?

[Andre]

dev,

I was wondering if anyone else is having trouble building NiFi after
purging their maven repositories?

$ git rebase apache/master
Current branch NIFI-871 is up to date.

$ mvn -U  -DskipTests=true  clean install
...
[WARNING] The POM for de.svenkubiak:jBcrypt:jar:0.4.1 is missing, no
dependency information available
...
[ERROR] Failed to execute goal on project nifi-standard-processors: Could
not resolve dependencies for project
org.apache.nifi:nifi-standard-processors:jar:1.2.0-SNAPSHOT: Could not find
artifact de.svenkubiak:jBcrypt:jar:0.4.1 in central (
https://repo1.maven.org/maven2) -> [Help 1]

>From what I gather the dependency has been renamed into jBCrypt?

Cheers

Re: [REMINDER] Please signoff when committing other people's changes

[Andre]

James,

There's no doubt the Sign-off-by is redundant (as GIT itself holds that
information, reason why GH is still able to show the information without
the sign-of-by stamp), however, I agree with your view around positive
action and easy to refer as Bryan pointed.

Joe,

Thanks for the clarification. If no-one opposes, I will update the
Contributor Guide regarding requirement vs. recommended as it seems to have
caused a small to confusion to some of the committers. If the message is
that consistency in this space is not required, than lets reflect this in
the documentation.

On a side note, it may be worth to note that a "+1 before merge" model
would sit in between CTR and RTC  - which technically seems to require
Consensus Approval (i.e. TTBOMK means 3 positive votes + lack of negative
votes in ASF lingo).

Formality of number aside, there's no doubt our model is working like a
charm! :-)

Cheers

On Fri, Mar 3, 2017 at 5:49 AM, James Wing  wrote:

> I recommend the practice.  Although the signoff may not be authoritative,
> it requires a positive action that suggests you purposefully merged the
> commit, as opposed to commits you might have accidentally merged and
> pushed.
>
> Thanks,
>
> James
>
>

[REMINDER] Please signoff when committing other people's changes

[Andre]

dev,

May I remind you to ensure we follow the Contributor Guide and use:

git commit --amend -s

when merging commits from your peers?

While git pretty-format can be used to reveal the committer, I am sure that
all of us will agree that as an inclusive community we value both the
pretty and ugly formats...

So can we give the ugly format the support it deserves and ensure we add
the neat Signed-off-by stamp to the commit message?

Cheers

Re: waitting for your hellp

[Andre]

My last message had some residue of copy and paste residues:

Where it reads:

"Using OpenTSDB's HTTP REST API together with the InvokeHTTP processor
(payload content must be processed to the desired format before reaching
PutTCP)"

it should read

"Using OpenTSDB's HTTP REST API together with the InvokeHTTP processor
(payload content must be processed to the desired format before reaching
InvokeHTTP)"

Cheers


On Mon, Feb 27, 2017 at 10:58 PM, Andre  wrote:

> Hi,
>
> Writing to OpenTSDB is a reasonably common pattern that can be achieved by:
>
> Using OpenTSDB's telnet API together with the PutTCP processor (payload
> content must be processed to the desired format before reaching PutTCP)
>
> or
>
> Using OpenTSDB's HTTP REST API together with the InvokeHTTP processor
> (payload content must be processed to the desired format before reaching
> PutTCP)
>
> Hope this help.
>
> Andre
>
> PS-Due to its (higher) number of subscribers, the user mailing list is
> usually a better channel for messages requesting guidance on use cases.
>
> On Mon, Feb 27, 2017 at 6:55 PM, sczylbh  wrote:
>
>> Hi，
>> I want to store the data to opentsdb or redis use nifi,Do you have any
>> good suggestions?
>> or can you develop the processor that can put/get data to/from opentsdb
>> or redis.
>
>
>

Re: waitting for your hellp

[Andre]

Hi,

Writing to OpenTSDB is a reasonably common pattern that can be achieved by:

Using OpenTSDB's telnet API together with the PutTCP processor (payload
content must be processed to the desired format before reaching PutTCP)

or

Using OpenTSDB's HTTP REST API together with the InvokeHTTP processor
(payload content must be processed to the desired format before reaching
PutTCP)

Hope this help.

Andre

PS-Due to its (higher) number of subscribers, the user mailing list is
usually a better channel for messages requesting guidance on use cases.

On Mon, Feb 27, 2017 at 6:55 PM, sczylbh  wrote:

> Hi，
> I want to store the data to opentsdb or redis use nifi,Do you have any
> good suggestions?
> or can you develop the processor that can put/get data to/from opentsdb or
> redis.

Re: Processors for Google Cloud Platform

[Andre]

Mikhail,

The definitive source would be

https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide#ContributorGuide-Providingcodeordocumentationcontributions

But it is more or less:

1. Search for an existing JIRA ticket covering the work
2. If none exists, touch base with the overall dev / user community (you
did that already)
3. Open the ticket and/or ask it to be assigned to you
4. Creating a git branch of the main NiFi tree (either via git or Github
https://github.com/apache/nifi )
5. Code! Code! Code!
6. Submit a patch (JIRA) or PR (Github)
7. Action on feedback provided during the peer view of your code until it
gets ready to be merged
8. Be proud when you commit get merged (and share on twitter that your
contribution has been immortalised by the git log. :-)  )

Cheers


PS-I also noticed that although you cross posted, my reply only went to
users, so shifting it back to dev as that is a more appropriate forum.


On Mon, Feb 27, 2017 at 3:30 PM, Mikhail Sosonkin 
wrote:

> Andre,
>
> I would love to get the PubSub processors into nifi proper. I'm not too
> familiar with your development processes, can you give me some directions
> on how you want me to do that and what sort of things the code should have?
>
> Thanks,
> Mike.
>
> On Sun, Feb 26, 2017 at 9:58 PM, Andre  wrote:
>
>> Mikhail,
>>
>> Thanks for the email!
>>
>> Someone had recently contributed processors to handle Google Cloud
>> Storage buckets and objects, however I am not aware of anyone providing
>> processors to handle PubSub yet so I am sure it will be a welcome processor!
>>
>> Would you be willing to submit a PR to include your processors as part of
>> the GCS bundle?
>>
>> Cheers
>>
>> On Mon, Feb 27, 2017 at 1:27 PM, Mikhail Sosonkin 
>> wrote:
>>
>>> Hello Everyone,
>>>
>>> SYNACK has recently started working with Google Cloud and we use NiFi to
>>> move some of our data sets. For that, we've built processors to work with
>>> Cloud Storage and Pubsub systems.
>>>
>>> A processor that uploads messages to Google Cloud Platform PubSub topic:
>>> https://github.com/synack/nifi-gcp-pubsub-publisher
>>>
>>> A processor that reads messages from a Google Cloud Platform PubSub
>>> topic: https://github.com/synack/nifi-gcp-pubsub-consumer
>>>
>>> A processor that uploads files to Google Cloud Platform Cloud: Storage:
>>> https://github.com/synack/nifi-gcp-putgcs
>>>
>>> They are certainly works in progress, but we're running them in our
>>> production environments inside and outside of GCP without any issues. Since
>>> these processors have been working so well for us, I wanted to share them
>>> with the community - just in case some one else finds them useful.
>>>
>>> Our team is still growing and being able to do these functions as part
>>> of NiFi, rather than building specialized widgets, has been a major boon
>>> for us. So, thanks NIFI team!
>>>
>>> Mike.
>>> http://synack.com
>>>
>>> This email may contain material that is confidential for the sole use of
>>> the intended recipient(s).  Any review, reliance or distribution or
>>> disclosure by others without express permission is strictly prohibited.  If
>>> you are not the intended recipient, please contact the sender and delete
>>> all copies of this message.
>>>
>>
>>
>
> This email may contain material that is confidential for the sole use of
> the intended recipient(s).  Any review, reliance or distribution or
> disclosure by others without express permission is strictly prohibited.  If
> you are not the intended recipient, please contact the sender and delete
> all copies of this message.
>

Quick question on LICENSE/NOTICE when bundling controllers and processors

[Andre]

Hi there,

Quick question on proper licensing:

When bundling Processors, Services and APIs, where should the NOTICES and
LICENSES be added to?

The PR in question is https://github.com/apache/nifi/pull/1541

My current reading is that all NAR levels will have to include the proper
references (although I may reduce a bit of the dependencies by excluding
some of the deeper dependencies, specially at the
nifi-irc-client-service-api-nar ).

Would you agree?

Cheers

Re: [DISCUSS] Scale-out/Object Storage - taming the diversity of processors

[Andre]

Joe,

Thanks for the comments!

Slightly deviating from code consistency discussion but I think you raised
some important points.

I may on the glass half empty side, but being a user who witnessed many
time how modularity played on other communities i am not particular excited
about the prospect.

I agree with Olegz comments about the benefits of the registry to code that
due to licensing issues would not be able to merged into NiFi, I am excited
about faster builds and selective packaging but past IMNHSO things get ugly
very quickly.

To be 100% transparent it is probably because I personally benefited from
having my code reviewed and subjected to the always helpful input of people
like yourself, Aldrin, BBende, JPercivall, Matt B, Oleg,  Pierre,  and the
rest of the wider community. I think of how ListenSMTP evolved in two days
more than I would be able to improve it in a lifetime, all thanks to a very
strong core of contributors around this community.

Perhaps it is because as a volunteer I have had to manage some sites build
around WordPress and noticed that quite frequently a WordPress site is a
see of plugin modules that as time progresses start getting outdated,
turning the long term maintenance of the platform an absolute pain. And to
be honest it has a healthy ecosystem around it: free modules, commercial
modules, independent registries, product reviews and support forums.

At this stage I would like to point out: Support forums - we need to start
thinking about how we plan to manage third party plugins as shared mailing
lists are woefully inadequate to do so. Even Elastic seems to have ended up
using both its support forums and GH's issue pages to provide support to
users.

In any case, back to the original issue:



I think I will second Adam's comments around not assuming the registry will
help around the original issue highlighted: Consistency, Code repetition
and long term support.



Cheers



On Thu, Feb 23, 2017 at 6:25 AM, Joe Witt  wrote:

> more good points...
>
> We will need to think through/document the benefits that this
> extension bazar would provide and explain the risks it also brings to
> users.  We should document for developers best practices in building
> new components or extending existing ones and we should allow users to
> socialize their findings for processors.  If someone sees a processor
> with 'one star' from a few users versus '4.5' from a thousand they
> would get a different level of confidence.  We should in general think
> about how to 'mature' components in these registries and such.
>
> That said, I think some of the problems we're talking about we'd be
> very honored and fortunate to be solving.  Right now we're making it
> hard to build and develop releases and slowing our agility as a
> community.
>
> Good problems to have though!
>
> joe
>
>

Re: [DISCUSS] Scale-out/Object Storage - taming the diversity of processors

[Andre]

Adam,

On 23 Feb 2017 4:43 AM, "Adam Lamar"  wrote:

Hey all,

I can understand Andre's perspective - when I was building the ListS3
processor, I mostly just copied the bits that made sense from ListHDFS and
ListFile. That worked, but its a poor way to ensure consistency across
List* processors.


Been there,  done that and continute to do that :-)

This is particularly tricky however,  because those processors drift apart
once the first iteration is made. Bugs get fixed in one processor but not
on the other.

As a once-in-a-while contributor, I love the idea that community
contributions are respected and we're not dropping them, because they solve
real needs right now, and it isn't clear another approach would be better.


I feel your pain and good news is that removing them is a breaking change
anyhow...

 plus we all love the *S3 processors.  :-)

And I disagree slightly with the notion that an artifact registry will
solve the problem - I think it could make it worse, at least from a
consistency point of view.


100% agreed.

Taming _is_ important, which is one reason
registry communities have official/sanctioned modules. Quality and
interoperability can vary vastly.


100% agreed. Just look at maven, jcenter, PyPI...


I suspect you will agree with idea that the user would think twice about
using a 3rd party processor due to the fear of it obsoleted by later
version upgrades?



By convention, it seems like NiFi already has a handful of well-understood
patterns - List, Fetch, Get, Put, etc all mean something specific in
processor terms. Is there a reason not to formalize those patterns in the
code as well? That would help with processor consistency, and if done
right, it may even be easier to write new processors, fix bugs, etc.


100% agreed. My suggestion was HCFS but our dislike for this approach
should not preclude us from achieving the final goal:

Currently consistency isn't easily maintained, it would be great if it did.

Thank lots for your comments, truly appreciated.

Re: [DISCUSS] Scale-out/Object Storage - taming the diversity of processors

[Andre]

Pierre,


> I believe NiFi is great for one reason: you have a lot of specialized
> processors that are really easy to use and efficient for what they've been
> designed for.
>

> Let's ask ourselves the question the other way: with the NiFi registry on
> its way, what is the problem having multiple processors for each back end?
> I don't really see the issue here. OK we have a lot of processors (but I
> believe this is a good point for NiFi, for user experience, for
> advertising, etc. - maybe we should improve the processor listing though,
> but again, this will be part of the NiFi Registry work), it generates a
> heavy NiFi binary (but that will be solved with the registry), but that's
> all, no?
>

The natural trade-off being fragmentation, code support and consistency?

Simple example?

ListS3 = Uses InputRequirement(Requirement.INPUT_FORBIDDEN)
ListGCSBucket = INPUT_FORBIDDEN seems to be absent, however, expression
language is disabled on most properties, suggesting design did not intend
to have input. Simple bug (NIFI-3514), simple fix (PR#1526).

Yes, no doubts, ListS3 presents S3's properties in clear fashion. Certainly
ListGCSBucket represents  GCS metadata as attributes in a more specific way
and this is handy, but that wouldn't be an unmanageable challenge.

This is not an isolated issue, there are plenty of examples, some as simple
as naming...  After all, one could be ultra pedantic for a second and note
the ListGCSBucket does not follow the same convention as ListS3(*).


Therefore, while the the examples above are overly trivial, they still
serve as a clear reminder of a very WET vs DRY dilemma. I strongly believe
we should strive to stay in DRY land.


Note however, that I am 100% OK with the idea that using HCFS may be overly
complex and possibly undesirable;

Nonetheless I think we should at least consider Matt's suggestion of using
some refactoring magic, or anything that can help us achieving programatic
ways of promoting consistency across the common features of those
processors (with the registry or not).



I will take the community guidance on this.

Cheers

Andre


(*) The closer conventional name would probably be ListGCS as no other
ListProcessor seems to define the unit of collection, (i.e. it is ListSFTP
not ListSFTPFolder).  I have not raised a JIRA ticket but I suggest the
name to be changed for better user experience.

Re: [DISCUSS] Scale-out/Object Storage - taming the diversity of processors

[Andre]

Andrew,


On Wed, Feb 22, 2017 at 11:21 AM, Andrew Grande  wrote:

> I am observing one assumption in this thread. For some reason we are
> implying all these will be hadoop compatible file systems. They don't
> always have an HDFS plugin, nor should they as a mandatory requirement.
>

You are partially correct.

There is a direct assumption in the availability of a HCFS (thanks Matt!)
implementation.

This is the case with:

* Windows Azure Blob Storage
* Google Cloud Storage Connector
* MapR FileSystem (currently done via NAR recompilation / mvn profile)
* Alluxio
* Isilon (via HDFS)
* others

But I would't say this will apply to every other use storage system and in
certain cases may not even be necessary (e.g. Isilon scale-out storage may
be reached using its native HDFS compatible interfaces).


Untie completely from the Hadoop nar. This allows for effective minifi
> interaction without the weight of hadoop libs for example. Massive size
> savings where it matters.
>
>
Are you suggesting a use case were MiNiFi agents interact directly with
cloud storage, without relying on NiFi hubs to do that?


> For the deployment, it's easy enough for an admin to either rely on a
> standard tar or rpm if the NAR modules are already available in the distro
> (well, I won't talk registry till it arrives). Mounting a common directory
> on every node or distributing additional jars everywhere, plus configs, and
> then keeping it consistent across is something which can be avoided by
> simpler packaging.
>

As long the NAR or RPM supports your use-case, which is not the case of
people running NiFi with MapR-FS for example. For those, a recompilation is
required anyway. A flexible processor may remove the need to recompile (I
am currently playing with the classpath implication to MapR users).

Cheers

Re: [DISCUSS] Scale-out/Object Storage - taming the diversity of processors

[Andre]

Andrew,

Thank you for contributing.

On 22 Feb 2017 10:21 AM, "Andrew Grande"  wrote:

Andre,

I came across multiple NiFi use cases where going through the HDFS layer
and the fs plugin may not be possible. I.e. when no HDFS layer present at
all, so no NN to connect to.


Not sure I understand what you mean.


Another important aspect is operations. Current PutHDFS model with
additional jar location, well, it kinda works, but I very much dislike it.
Too many possibilities for a human error in addition to deployment pain,
especially in a cluster.


Fair enough. Would you mind expanding a bit on what sort of  challenges
currently apply in terms of cluster deployment?


Finally, native object storage processors have features which may not even
apply to the HDFS layer. E.g. the Azure storage has Table storage, etc.


This is a very valid point but I am sure exceptions (in this case a NoSQL
DB operating under the umbrella term of "storage").

I perhaps should have made it more explicit but the requirements are:

- existence of a hadoop compatible interface
- ability to handle files

Again, thank you for the input, truly appreciated.

Andre

I agree consolidating various efforts is worthwhile, but only within a
context of a specific storage solution. Not 'unifying' them into a single
layer.

Andrew

On Tue, Feb 21, 2017, 6:10 PM Andre  wrote:

> dev,
>
> I was having a chat with Pierre around PR#379 and we thought it would be
> worth sharing this with the wider group:
>
>
> I recently noticed that we merged a number of PRs and merges around
> scale-out/cloud based object store into the master.
>
> Would it make sense to start considering adopting a pattern where
> Put/Get/ListHDFS are used in tandem with implementations of the
> hadoop.filesystem interfaces instead of creating new processors, except
> where a particular deficiency/incompatibility in the hadoop.filesystem
> implementation exists?
>
> Candidates for removal / non merge would be:
>
> - Alluxio (PR#379)
> - WASB (PR#626)
>  - Azure* (PR#399)
> - *GCP (recently merged as PR#1482)
> - *S3 (although this has been in code so it would have to be deprecated)
>
> The pattern would be pretty much the same as the one documented and
> successfully deployed here:
>
> https://community.hortonworks.com/articles/71916/connecting-
> to-azure-data-lake-from-a-nifi-dataflow.html
>
> Which means that in the case of Alluxio, one would use the properties
> documented here:
>
> https://www.alluxio.com/docs/community/1.3/en/Running-
> Hadoop-MapReduce-on-Alluxio.html
>
> While with Google Cloud Storage we would use the properties documented
> here:
>
> https://cloud.google.com/hadoop/google-cloud-storage-connector
>
> I noticed that specific processors could have the ability to handle
> particular properties to a filesystem, however I would like to believe the
> same issue would plague hadoop users, and therefore is reasonable to
> believe the Hadoop compatible implementations would have ways of exposing
> those properties as well?
>
> In the case the properties are exposed, we perhaps simply adjust the *HDFS
> processors to use dynamic properties to pass those to the underlying
> module, therefore providing a way to explore particular settings of an
> underlying storage platforms.
>
> Any opinion would be welcome
>
> PS-sent it again with proper subject label
>

[DISCUSS] Scale-out/Object Storage - taming the diversity of processors

[Andre]

dev,

I was having a chat with Pierre around PR#379 and we thought it would be
worth sharing this with the wider group:


I recently noticed that we merged a number of PRs and merges around
scale-out/cloud based object store into the master.

Would it make sense to start considering adopting a pattern where
Put/Get/ListHDFS are used in tandem with implementations of the
hadoop.filesystem interfaces instead of creating new processors, except
where a particular deficiency/incompatibility in the hadoop.filesystem
implementation exists?

Candidates for removal / non merge would be:

- Alluxio (PR#379)
- WASB (PR#626)
 - Azure* (PR#399)
- *GCP (recently merged as PR#1482)
- *S3 (although this has been in code so it would have to be deprecated)

The pattern would be pretty much the same as the one documented and
successfully deployed here:

https://community.hortonworks.com/articles/71916/connecting-
to-azure-data-lake-from-a-nifi-dataflow.html

Which means that in the case of Alluxio, one would use the properties
documented here:

https://www.alluxio.com/docs/community/1.3/en/Running-
Hadoop-MapReduce-on-Alluxio.html

While with Google Cloud Storage we would use the properties documented here:

https://cloud.google.com/hadoop/google-cloud-storage-connector

I noticed that specific processors could have the ability to handle
particular properties to a filesystem, however I would like to believe the
same issue would plague hadoop users, and therefore is reasonable to
believe the Hadoop compatible implementations would have ways of exposing
those properties as well?

In the case the properties are exposed, we perhaps simply adjust the *HDFS
processors to use dynamic properties to pass those to the underlying
module, therefore providing a way to explore particular settings of an
underlying storage platforms.

Any opinion would be welcome

PS-sent it again with proper subject label

Scale-out/Object Storage - taming the deviersity of processors

[Andre]

dev,

I was having a chat with Pierre around PR#379 and we thought it would be
worth sharing this with the wider group:


I recently noticed that we merged a number of PRs and merges around
scale-out/cloud based object store into the master.

Would it make sense to start considering adopting a pattern where
Put/Get/ListHDFS are used in tandem with implementations of the
hadoop.filesystem interfaces instead of creating new processors, except
where a particular deficiency/incompatibility in the hadoop.filesystem
implementation exists?

Candidates for removal / non merge would be:

- Alluxio (PR#379)
- WASB (PR#626)
 - Azure* (PR#399)
- *GCP (recently merged as PR#1482)
- *S3 (although this has been in code so it would have to be deprecated)

The pattern would be pretty much the same as the one documented and
successfully deployed here:

https://community.hortonworks.com/articles/71916/connecting-to-azure-data-lake-from-a-nifi-dataflow.html

Which means that in the case of Alluxio, one would use the properties
documented here:

https://www.alluxio.com/docs/community/1.3/en/Running-Hadoop-MapReduce-on-Alluxio.html

While with Google Cloud Storage we would use the properties documented here:

https://cloud.google.com/hadoop/google-cloud-storage-connector

I noticed that specific processors could have the ability to handle
particular properties to a filesystem, however I would like to believe the
same issue would plague hadoop users, and therefore is reasonable to
believe the Hadoop compatible implementations would have ways of exposing
those properties as well?

In the case the properties are exposed, we perhaps simply adjust the *HDFS
processors to use dynamic properties to pass those to the underlying
module, therefore providing a way to explore particular settings of an
underlying storage platforms.

Any opinion would be welcome

Re: [ANNOUNCE] New Apache NiFi Committer Jeff Storck

[Andre]

Welcome aboard Jeff! Well deserved

On Wed, Feb 22, 2017 at 6:41 AM, Aldrin Piri  wrote:

> On behalf of the Apache NiFI PMC, I am very pleased to announce that Jeff
> Storck has accepted the PMC's invitation to become a committer on the
> Apache NiFi project. We greatly appreciate all of Jeff's hard work and
> generous contributions and look forward to continued involvement in the
> project.
>
> Jeff's contributions include significant efforts toward upgrade and
> migration processes inclusive of flow layout when upgrading from 0.x to 1.x
> and the ZooKeeper migration toolkit.
>
> Congrats, Jeff!
>

Re: Starting apache NiFi as a windows service

[Andre]

Joey,

At the moment I am experimenting with

https://github.com/kohsuke/winsw



On 16 Feb 2017 01:53, "iCloud"  wrote:

Andre, can you share any details on how you’re approaching this? Are you
using an existing service wrapper? The javapackager tool added in Java 8?
Etc.

On Feb 14, 2017, 8:50 PM -0600, Andre , wrote:
> Chris,
>
> Not that I am aware of but I am working on this atm. Should have something
> baked until the weekend.
>
> On Wed, Feb 15, 2017 at 12:25 PM, Chris Bulleri  wrote:
>
> > Are there any instructions for starting the latest apache NiFi as a
windows
> > service? Server 2012 in particular.
> >
> >
> > Chris Bulleri
> > 2HB Inc.
> > Principal/CTO
> > ch...@2hb.com
> > Voice: 410.872.2800 x717
> >

Re: travis-ci auto cancellation

[Andre]

Aldrin,

The feature is enabled and seems to be working greatly. The only thing I am
keeping an eye on are overly extensive delays due to resource allocation. I
suspect once a job gets cancelled, the next one gets shoved to the end of
the queue.

Depending on my findings I may feed the travis folks with some suggestions.

Let me know if you have issues.

On Sun, Feb 19, 2017 at 12:15 PM, Aldrin Piri  wrote:

> Sounds like a superb feature to enable.  Anything we can do to be more
> effective with our Travis cycles is most welcomed.
>
> On Fri, Feb 17, 2017 at 8:22 AM, Andre  wrote:
>
> > dev,
> >
> > Would you mind if I configured the NiFi, MiNiFi and MiNiFI-CPP travis-ci
> > integration to perform auto-cancellation when new commits are pushed to
> the
> > same branch or PR?
> >
> > This should significantly reduce the number of builds we do per busy days
> > and offer benefits to the whole community.
> >
> > Cheers
> >
>

Release trick: Checking release source packages with git

[Andre]

dev,

When voting on a release I attempt to verify the source packages signed by
the release manager and the commit ID listed as part of the VOTE
announcement.

For a long time I found the process a painful step - albeit important -
 that required all sort of weird bash-fu.

Anyhow., I suspect that I found a way - which some of you may already know
- to do that using GIT and decided to share here so it can be validated by
others:


ASSUMPTIONS:

NiFi repo (linked to git) =>  /home/username/nifi

unzipped release files =>  /home/username/temp/nifi-release-version

PROCEDURE:

1. ensure the main repo is set to the correct commit:

$ cd /home/username/nifi
$ git checkout commit_id

2. change to the unziped source directory

$ cd /home/username/temp/nifi-release-version

3. validate the sources:

$ git --git-dir=/home/username/nifi/.git status --short | grep -v
".gitignore$"
?? DEPENDENCIES

4. If you are feeling paranoid, you can further validate the command by
introducing changes to a file

$ echo AAA >> nifi-assembly/pom.xml

And re-running the command:

$ git --git-dir=/home/username/nifi/.git status --short | grep -v
".gitignore$"
 M nifi-assembly/pom.xml
?? DEPENDENCIES

5. Continue review of the release with the piece of mind offerded by
knowing the Release Manager has not introduced adware to the source
packages. 8-D


For more information:

https://git-scm.com/blog/2010/04/11/environment.html


Cheers

Re: [VOTE] Release Apache NiFi 0.7.2

[Andre]

+1 binding.

On Sat, Feb 18, 2017 at 6:51 AM, Matt Gilman 
wrote:

> +1 (binding) Release this package as nifi-0.7.2
>
> On Thu, Feb 16, 2017 at 11:42 PM, Andy LoPresto 
> wrote:
>
> > Hello,
> >
> > I am pleased to be calling this vote for the source release of Apache
> NiFi
> > nifi-0.7.2.
> >
> > The source zip, including signatures, digests, etc. can be found at:
> > https://repository.apache.org/content/repositories/orgapachenifi-1099
> >
> > The Git tag is nifi-0.7.2-RC1
> > The Git commit ID is 831ac6939df7d418cadb336eedb4e09fb97541a1
> > https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> > 831ac6939df7d418cadb336eedb4e09fb97541a1
> >
> > Checksums of nifi-0.7.2-source-release.zip:
> > MD5: efe9ea1cf0698a57a6829fe3829fc136
> > SHA1: aee51164af34394c7017dae491b239a88b614865
> > SHA256: 8cd02675003fca837ea0092b6225600a4700b905e5214a751ae7ff833263193b
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/alopresto.asc
> >
> > KEYS file available here:
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > 2 issues were closed/resolved for this release:
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > version=12339601&projectId=12316020
> >
> > Release note highlights can be found here:
> > https://cwiki.apache.org/confluence/display/NIFI/
> > Release+Notes#ReleaseNotes-Version0.7.2
> >
> > The vote will be open for 96 hours (over holiday weekend).
> > Please download the release candidate and evaluate the necessary items
> > including checking hashes, signatures, build
> > from source, and test.  The please vote:
> >
> > [ ] +1 Release this package as nifi-0.7.2
> > [ ] +0 no opinion
> > [ ] -1 Do not release this package because because…
> >
> > Andy LoPresto
> > alopre...@apache.org
> > *alopresto.apa...@gmail.com *
> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >
> >
>

Re: [VOTE] Release Apache NiFi 1.1.2

[Andre]

+1 - release

On Fri, Feb 17, 2017 at 2:37 PM, Andy LoPresto  wrote:

> Hello,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> nifi-1.1.2.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1098
>
> The Git tag is nifi-1.1.2-RC1
> The Git commit ID is 51fad01f5daf33716b8b5379c32ee932d91c8c63
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> 51fad01f5daf33716b8b5379c32ee932d91c8c63
>
> Checksums of nifi-1.1.2-source-release.zip:
> MD5: 31d20a09955fac32d5b4147c497deede
> SHA1: 9f8f2ce00397d990dfb0890f9b1ede70dca4f25e
> SHA256: 0f57b5ae7f5e1d7f1289d779df39f20d70af0ffd92cb01b265beb90b309b747e
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/alopresto.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 2 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> version=12339600&projectId=12316020
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-Version1.1.2
>
> The vote will be open for 96 hours (over holiday weekend).
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test.  The please vote:
>
> [ ] +1 Release this package as nifi-1.1.2
> [ ] +0 no opinion
> [ ] -1 Do not release this package because because…
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
>

travis-ci auto cancellation

[Andre]

dev,

Would you mind if I configured the NiFi, MiNiFi and MiNiFI-CPP travis-ci
integration to perform auto-cancellation when new commits are pushed to the
same branch or PR?

This should significantly reduce the number of builds we do per busy days
and offer benefits to the whole community.

Cheers

NIFI-329 - IRC Processors

[Andre]

dev,

I am starting to work on NIFI-329 and plan to soon introduce processors
capable of interacting with IRC servers.

At this stage I am looking to implement only PublishIRC, however, given the
processor will use a controller service this may be later on expanded to
handle IRC message consumption (i.e. ConsumeIRC) and perhaps bot type
functionality by mimicking the ethos of HandleHttpRequest and
HandleHttpResponse processors.

Since this is the first time I am trying to write a controller service may
I ask you some feedback about my first reading?

Currently the "best" ASL2 IRC library seems to be KICL (Kitteh IRC Client
library[1]) which happens to support IRCv3, SSL and a bunch of other stuff.

KICL Client class seems to fit very well within a controller service
onEnabled, but I have the impression its event driven API is not a natural
fit to NiFi processors and will require some work similar to the one
required to get ListenSMTP working properly.

Is this reading correct? In you experience, is it manageable or should I
look for another library?


Any feedback will be appreciated.


[1] -
http://kicl.kitteh.org/en/latest/#kitteh-irc-client-library-documentation

Re: Moving ZIP and TAR.GZ "formats" into an explicit profile

[Andre]

All,

This is a reminder that the dir-only profile has been merged to the master
branch. So for those trying to save SSD writes or simply skip a few CPU
cycles archiving the nifi assembly during development, you can now use mvn
-Pdir-only to skip the pain.

I also would like to point out that since
commit 7e97946c359bfcb62ef39a2ce37083554501f1b0, parallel test runs now
seem to be far more stable. If things continue stable, we may later on
re-introduce parallel test runs to travis.

Cheers



On Fri, Feb 3, 2017 at 11:04 PM, Andre  wrote:

> devs,
>
> Thank you for the comments.
>
> NIFI-3434 has been raised and submitted as PR#1472. The commit introduces
> generateArchives (default and [hopefully] mimicking current behavior) and
> dir-only (skipping the archive generation).
>
> Once again thank you for your comments.
>
> On Fri, Feb 3, 2017 at 10:34 AM, Andy LoPresto 
> wrote:
>
>> +1 to Russell’s point and thanks Andre for bringing up the topic. I’d
>> hesitate to change the default now because people probably are depending on
>> the default, but I wouldn’t object to a shift in the default at a major
>> release.
>>
>> Andy LoPresto
>> alopre...@apache.org
>> *alopresto.apa...@gmail.com *
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On Feb 2, 2017, at 8:27 AM, Russell Bateman 
>> wrote:
>>
>> Maven profiles could be used to sort this out, one for just the zip,
>> another for just the tarball and a third, maybe the default, to continue
>> working the way it has been working.
>>
>> On 02/02/2017 09:09 AM, Aldrin Piri wrote:
>>
>> I think this could be useful.  Only caveat is that I'm sure there are
>> folks
>> in the community that have automated processes that make use of these
>> binaries.
>>
>> From the dev standpoint, I could see a profile that disables the assembly
>> from happening such that the build occurs as it does now unless folks
>> explicitly want to avoid it.  Regardless of implementation, can see why it
>> would be helpful.
>>
>> On Thu, Feb 2, 2017 at 9:29 AM, Andre  wrote:
>>
>> devs,
>>
>> Currently calling 'mvn clean install' creates a ZIP, a TAR.GZ and a
>> directory containing the same code. This leads to wasted disk space and a
>> lot of wasted disk writes (something that a lot of folks using SSDs prefer
>> to avoid).
>>
>> Would anyone oppose the idea of moving the ZIP and TAR.GZ assemblies into
>> a
>> "release" profile (or whatever name we agree to). This way we could
>> maintain the directory "format" (which I suspect most of us use during
>> development), while still providing a convenient way of creating the ZIP
>> and TAR.GZ packages.
>>
>>
>> Depending on the feedback I will be happy to raise the JIRA and work on
>> it.
>>
>> Cheers
>>
>>
>>
>>
>

Re: Starting apache NiFi as a windows service

[Andre]

Chris,

Not that I am aware of but I am working on this atm. Should have something
baked until the weekend.

On Wed, Feb 15, 2017 at 12:25 PM, Chris Bulleri  wrote:

> Are there any instructions for starting the latest apache NiFi as a windows
> service?  Server 2012 in particular.
>
>
> Chris Bulleri
> 2HB Inc.
> Principal/CTO
> ch...@2hb.com
> Voice: 410.872.2800 x717
>

Re: MiNiFi Target audience

[Andre]

Marc,

I would say that lack IPv6 support was not intentional and certainly in
favor of addressing it.

Cheers


On Tue, Feb 14, 2017 at 6:00 AM, Marc P.  wrote:

> Good Afternoon,
>Given the plethora of devices that MiNiFi may be installed onto, would
> it make sense to ensure we support IPv4 and IPv6?
>
>  For example, when searching for IPv4 tickets I found the following JIRA
> ticket that indicates we'll support IPv4. I'm abstracting some of the
> socket communications into an abstract facade to facilitate unit and
> integration testing more easily -- and simply to consolidate the code into
> a more maintainable format.
> https://issues.apache.org/jira/browse/MINIFI-159
>
> Upon seeing this the code will not work with IPv6. Is this intentional?
> Should we change this to support either? We can retrieve the hostname
> regardless of the internet protocol version.
>
> Without further input my intent would be change to using getaddrinfo(...)
> withint MiNiFi-cpp. Would love to hear a discussion on whether making this
> change goes against the intent of the software.
>
> While we may not support IPv6 now, the code that exists now is more
> resilient, and is therefore on my radar to reduce artifacts.
>
>Thanks,
>Marc "aka. Marc"
>

Re: PR Builds

[Andre]

Joe,

Perhaps we should float the following feature to the Apache INFRA folks?

https://github.com/travis-ci/beta-features/issues/3


I am sure most ASF projects would incredibly benefit from travis queue
deduping.

Cheers

On Sat, Feb 11, 2017 at 7:49 AM, Joe Witt  wrote:

> Otto,
>
> We get fairly sketchy results with Travis due to build time/timeouts
> as it is a very limited infrastructure.  We're evaluating ways to make
> that more reliable because passing PR/builds do help ease PR review.
> The appveyor builds are always broken at this point.
>
> But, you're PR is in and will get reviewed.  It is really interesting
> that you've run into these osx issues.  Many of us build on OSX
> non-stop so it is curious but we'll check it out.
>
> Thanks for contributing!
>
> Joe
>
> On Fri, Feb 10, 2017 at 3:38 PM, Otto Fowler 
> wrote:
> > Hi,
> >
> > I submitted my first PR today, but I’m confused about the build status.
> > Every PR against the project looks like it is failed in some way. Is
> there
> > an explanation for this?
>

Re: [VOTE] Establish Registry, a sub-project of Apache NiFi

[Andre]

+1 binding

On Sat, Feb 11, 2017 at 3:40 AM, Bryan Bende  wrote:

> All,
>
> Following a solid discussion for the past few days [1] regarding the
> establishment of Registry as a sub-project of Apache NiFi, I'd like to
> call a formal vote to record this important community decision and
> establish consensus.
>
> The scope of this project is to define APIs for interacting with
> resources that one or more NiFi instances may be interested in, such
> as a flow registry for versioned flows, an extension registry for
> extensions, and possibly other configuration resources in the future.
> In addition, this project will provide reference implementations of
> these registries, with the goal of allowing the community to build a
> diverse set of implementations, such as a Git provider for versioned
> flows, or a bintray provider for an extension registry.
>
> I am a +1 and looking forward to the future work in this area.
>
> The vote will be open for 72 hours and be a majority rule vote.
>
> [ ] +1 Establish Registry, a subproject of Apache NiFi
> [ ]   0 Do not care
> [ ]  -1 Do not establish Registry, a subproject of Apache NiFi
>
> Thanks,
>
> Bryan
>
> [1] http://mail-archives.apache.org/mod_mbox/nifi-dev/201702.mbox/%3CCALo_
> M19euo2LLy0PVWmE70FzeLhQRcCtX6TC%3DqoiBVfn4zFQMA%40mail.gmail.com%3E
>

[NIFI-1657] Heads-up: NiFi's travis-ci is going "multi-lingual"

[Andre]

devs,

In order to improve our ability to detect locale related bugs during
development, Pierre and I have been working on getting the surefire-plugin
to support user defined languages and to add some of these languages to the
travis-ci environment matrix.

It took a bit of trial and error but it seems like we are finally on the
way to merge.

Once the PR gets merged, travis-ci will start building using the following
locales:

* en_US;
* ja_JP;
* pt_BR, and
* fr_FR

the locale can also be set on you local development environment - without
changes to your overall operating system - by using the following
properties:

mvn -Pcontrib-check verify -Dsurefire.user.language=hu
-Dsurefire.user.region=HU


We know that at this stage some of the tests will fail but hopefully the
community will soon manage to address those.

Cheers

Re: C++ Style Guide

[Andre]

Aldrin,

I am also aware of someone trying to use googletest to streamline the
creation of cpp test coverage. I believe we may see a PR sometime later
this week.

Cheers

On Tue, Feb 7, 2017 at 8:51 AM, Aldrin Piri  wrote:

> Marc,
>
> I think that makes a lot of sense and certainly would be a +1 for the
> effort.
>
> Would it also make sense to roll in the associated linter [1] and make that
> an available target of build process to help identify compliance?
>
> [1] https://github.com/google/styleguide/tree/gh-pages/cpplint
>
> On Mon, Feb 6, 2017 at 2:51 PM, Marc  wrote:
>
> > Hello Everyone,
> >
> >   I'm submitting a few PRs to the MiNiFi CPP project and would like to
> > discuss a style guide for the C++ development. I'm partial to Google's
> > Style guide:
> >
> > https://google.github.io/styleguide/cppguide.html
> >
> >   Are there any objections to this? As I make PRs I'd like to make some
> > changes in favor of this style guide and wanted some support for or
> against
> > this. I don't intend on making wholesale changes for the sake of the
> style
> > guide, but instead will make these changes iteratively with my PRs.
> >
> >   In the future if there is support for it we can create a pre-commit
> hook
> > to 'beautify the code' with a tool like clang-format or astyle.
> >
> >   Thoughts for/against this style guide?
> >
> >   Thanks,
> >   Marc
> >
>

Re: PutFTP Port, Username and Password are EL evaluated but properties lack expressionLanguageSupported(true)

[Andre]

On Tue, Feb 7, 2017 at 12:40 AM, Mark Payne  wrote:

> Andre,
>
> "This design choice seems to trigger" -- I don't think this is a design
> choice but rather, a bug :)
>

Given you are the authoritative source I will trust your assessment.
NIFI-3442


>
> The Property Descriptor should indicate that Expression Language is
> supported. The mock framework is setup
> to fail the unit test because the processor's behavior does not match its
> documentation (i.e., it does in fact evaluate
> the Expression Language, but its property descriptor says that it does
> not). So I believe the fix is to simply update
> the Property Descriptor to indicate that it does in fact support
> Expression Language.
>
>
Yes. That is what I had done, will submit the PR shortly.

Cheers

PutFTP Port, Username and Password are EL evaluated but properties lack expressionLanguageSupported(true)

[Andre]

devs,


As part of NIFI-940 I am crafting test coverage for the PutFTP processor.
Upon testing I noticed the following discrepancy:

The port property lacks Expression Language Support

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/FTPTransfer.java#L71

while client.connect seems to evaluate the property.

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/FTPTransfer.java#L531

This design choice seems to trigger

java.lang.IllegalStateException: Attempting to Evaluate Expressions but
PropertyDescriptor[Username] indicates that the Expression Language is not
supported. If you realize that this is the case and do not want this error
to occur, it can be disabled by calling
TestRunner.setValidateExpressionUsage(false)


Is this normal?

Reading the code I suspect the reason for that is the intention to allow
flowfiles to contain the username, password and port attributes and
dynamically route them, but if that's the case, shouldn't we simply mark
the affected properties as supporting expression language?

Cheers

PutFile,PutHDFS handle similar properties differently

[Andre]

devs,

I was working on NIFI-940 when I noticed that PutFile and PutHDFS, while
similar in nature have different ways of handling some of its common
properties, more precisely, permissions (there may be others).

While PutHDFS validates permissions via custom validators[1], PutFile will
only validate permissions within onTrigger and if invalid permissions are
configured, will just log an event.[1]

I suppose that changing this behavior would break compatibility and will
have to wait for a major release, but would others agree this issue should
be addressed?

Cheers


[1]
https://github.com/apache/nifi/blob/7f5eabd603bfc326dadc35590bbe69304e8c90fa/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java#L145

[2]
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutFile.java#L263

Re: Moving ZIP and TAR.GZ "formats" into an explicit profile

[Andre]

devs,

Thank you for the comments.

NIFI-3434 has been raised and submitted as PR#1472. The commit introduces
generateArchives (default and [hopefully] mimicking current behavior) and
dir-only (skipping the archive generation).

Once again thank you for your comments.

On Fri, Feb 3, 2017 at 10:34 AM, Andy LoPresto  wrote:

> +1 to Russell’s point and thanks Andre for bringing up the topic. I’d
> hesitate to change the default now because people probably are depending on
> the default, but I wouldn’t object to a shift in the default at a major
> release.
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Feb 2, 2017, at 8:27 AM, Russell Bateman  wrote:
>
> Maven profiles could be used to sort this out, one for just the zip,
> another for just the tarball and a third, maybe the default, to continue
> working the way it has been working.
>
> On 02/02/2017 09:09 AM, Aldrin Piri wrote:
>
> I think this could be useful.  Only caveat is that I'm sure there are folks
> in the community that have automated processes that make use of these
> binaries.
>
> From the dev standpoint, I could see a profile that disables the assembly
> from happening such that the build occurs as it does now unless folks
> explicitly want to avoid it.  Regardless of implementation, can see why it
> would be helpful.
>
> On Thu, Feb 2, 2017 at 9:29 AM, Andre  wrote:
>
> devs,
>
> Currently calling 'mvn clean install' creates a ZIP, a TAR.GZ and a
> directory containing the same code. This leads to wasted disk space and a
> lot of wasted disk writes (something that a lot of folks using SSDs prefer
> to avoid).
>
> Would anyone oppose the idea of moving the ZIP and TAR.GZ assemblies into a
> "release" profile (or whatever name we agree to). This way we could
> maintain the directory "format" (which I suspect most of us use during
> development), while still providing a convenient way of creating the ZIP
> and TAR.GZ packages.
>
>
> Depending on the feedback I will be happy to raise the JIRA and work on it.
>
> Cheers
>
>
>
>

Moving ZIP and TAR.GZ "formats" into an explicit profile

[Andre]

devs,

Currently calling 'mvn clean install' creates a ZIP, a TAR.GZ and a
directory containing the same code. This leads to wasted disk space and a
lot of wasted disk writes (something that a lot of folks using SSDs prefer
to avoid).

Would anyone oppose the idea of moving the ZIP and TAR.GZ assemblies into a
"release" profile (or whatever name we agree to). This way we could
maintain the directory "format" (which I suspect most of us use during
development), while still providing a convenient way of creating the ZIP
and TAR.GZ packages.


Depending on the feedback I will be happy to raise the JIRA and work on it.

Cheers

Re: [ANNOUNCE] New Apache NiFi PMC Member - Joe Skora

[Andre]

Congratulations Joe!

On Fri, Dec 30, 2016 at 5:02 AM, Aldrin Piri  wrote:

> All,
>
> On behalf of the Apache NiFi PMC, I am pleased to announce that Joe Skora has
> accepted the PMC's invitation to join the Apache NiFi PMC.  Joe has been
> with NiFi for quite some time, even before its arrival in the ASF and
> became a committer in February.  We are most pleased he brought his
> knowledge and supported the community once open sourced and has has
> provided continuous and excellent contributions in all facets of the
> community.  Of note, Joe was our first community member to carry out a
> release without being a PMC member for 0.7.1.
>
> Please join us in congratulating and welcoming Joe to the Apache NiFi PMC.
>
> Again, congratulations Joe and well deserved!
>