Re: [ANNOUNCE] New NiFi PMC member Jeremy Dyer

[Jeff]

Congrats, Jeremy!

On Thu, Aug 2, 2018 at 10:06 AM Jeremy Dyer  wrote:

> I am very excited to be more involved in such a great community! Thank you
> everyone for the kind words and looking forward to what the future brings
> in this community!
>
> Thanks!
> Jeremy Dyer
>
> On Wed, Aug 1, 2018 at 8:46 AM, Yolanda Davis 
> wrote:
>
> > Congratulations Jeremy!
> >
> > On Wed, Aug 1, 2018 at 7:23 AM, Rob Moran  wrote:
> >
> > > Congratulations, Jeremy!
> > >
> > > On Tue, Jul 31, 2018 at 1:17 PM Sivaprasanna <
> sivaprasanna...@gmail.com>
> > > wrote:
> > >
> > > > Congratulations, Jeremy. Well deserved.
> > > >
> > > > On Tue, 31 Jul 2018 at 9:44 PM, Scott Aslan 
> > > wrote:
> > > >
> > > > > Congrats Jeremy!
> > > > >
> > > > > On Tue, Jul 31, 2018 at 11:02 AM Michael Moser  >
> > > > wrote:
> > > > >
> > > > > >  Congrats, welcome, and thank you for your work.
> > > > > >
> > > > > >
> > > > > > On Tue, Jul 31, 2018 at 9:21 AM Otto Fowler <
> > ottobackwa...@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Congratulations!
> > > > > > >
> > > > > > >
> > > > > > > On July 31, 2018 at 08:36:48, Tony Kurc (tk...@apache.org)
> > wrote:
> > > > > > >
> > > > > > > All,
> > > > > > >
> > > > > > > On behalf of the Apache NiFi PMC, I am pleased to announce that
> > > > Jeremy
> > > > > > Dyer
> > > > > > > has accepted the PMC's invitation to join the Apache NiFi PMC.
> > > > > > >
> > > > > > > Jeremy has been a long-time contributor to the project - across
> > > many
> > > > > > > different parts of the project to include NiFi and and MiNiFi,
> > > > > > contributing
> > > > > > > code, reviews, release verification, and help on the mailing
> > lists.
> > > > > He's
> > > > > > > performed the challenging, detail oriented work of acting as a
> > > > release
> > > > > > > manager for both the Java and C++ versions of MiNiFi (0.5.0 of
> > > each).
> > > > > > >
> > > > > > > Congratulations Jeremy and well deserved!
> > > > > > >
> > > > > > > Tony
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > --
> > yolanda.m.da...@gmail.com
> > @YolandaMDavis
> >
>

Re: [ANNOUNCE] New NiFi PMC member Kevin Doran

[Jeff]

Congrats Kevin!

On Wed, Aug 1, 2018 at 10:07 AM Kevin Doran  wrote:

> Thanks, everyone! There are so many exciting things going on in the Apache
> NiFi project and community these days. It's really fun to be involved and a
> great opportunity to work with such a terrific community! I'm happy to
> contribute in any way that I can.
>
> Thanks!
> Kevin
>
>
> On Wed, Aug 1, 2018 at 8:47 AM, Yolanda Davis 
> wrote:
>
> > Congratulations Kevin!
> >
> > On Wed, Aug 1, 2018 at 7:23 AM, Rob Moran  wrote:
> >
> > > Congratulations, Kevin!
> > >
> > > On Tue, Jul 31, 2018 at 1:16 PM Sivaprasanna <
> sivaprasanna...@gmail.com>
> > > wrote:
> > >
> > > > Congratulations, Kevin.
> > > >
> > > > On Tue, 31 Jul 2018 at 9:43 PM, Scott Aslan 
> > > wrote:
> > > >
> > > > > Congrats Kevin!
> > > > >
> > > > > On Tue, Jul 31, 2018 at 11:01 AM Michael Moser  >
> > > > wrote:
> > > > >
> > > > > > Congrats, welcome, and thank you for your work.
> > > > > >
> > > > > >
> > > > > > On Tue, Jul 31, 2018 at 9:21 AM Otto Fowler <
> > ottobackwa...@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Congratulations!
> > > > > > >
> > > > > > >
> > > > > > > On July 31, 2018 at 08:26:34, Tony Kurc (tk...@apache.org)
> > wrote:
> > > > > > >
> > > > > > > NiFi Community,
> > > > > > >
> > > > > > > On behalf of the Apache NiFi PMC, I am pleased to announce that
> > > Kevin
> > > > > > Doran
> > > > > > > has accepted the PMC's invitation to join the Apache NiFi PMC.
> > > > > > >
> > > > > > > In addition to being a regular code contributor to the project
> > for
> > > > > quite
> > > > > > > some time, Kevin has been hard to miss on the mailing lists,
> > > > especially
> > > > > > on
> > > > > > > NiFi Registry threads. We all appreciate his hard work getting
> > > > releases
> > > > > > > "out the door", helping verify releases and recently doing
> > release
> > > > > > manager
> > > > > > > duty for the NiFi Registry 0.2.0.
> > > > > > >
> > > > > > > Please join us in congratulating and welcoming Kevin to the
> > Apache
> > > > NiFi
> > > > > > > PMC.
> > > > > > >
> > > > > > > Tony
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > --
> > yolanda.m.da...@gmail.com
> > @YolandaMDavis
> >
>

Re: Java 11 is now ready

[Jeff]

Hi Mike,

I've been working on NIFI-5174 [1], which has a few subtasks for getting
NiFi ready for Java 9+.  Now that Java 11 is out officially, I'll start
targeting that version.  For building on Java 9+, it's fairly close to
being done.  It's been an interesting experience moving past Java 8, so far!

[1] https://issues.apache.org/jira/browse/NIFI-5174

On Tue, Sep 25, 2018 at 2:30 PM Mike Thomsen  wrote:

> For anyone interested in a breakdown:
>
> https://www.infoq.com/news/2018/09/java11-released
>
> As it is the first real LTS release post Java 8, we'll probably want to
> start experimenting with it in the near future.
>

Re: How does logging level get set?

[Jeff]

One point for using the isDebugEnabled (or any of the other level-checking
methods) is that if, in the values being logged, there is string
concatenation happening in the log.{level} statement itself. The cost of
that is paid whether the level is enabled or not.  Even with parameterized
logging, if the log.{level} call is passed a method invocation as a
parameter, the cost is paid there as well, as Kevin mentioned.

The best rule of thumb, IMO, is to use the enabled check if a parameter or
result of string concatenation with something being logged will cause
computation that is "expensive", such as sorting a list, especially if the
logging would be invoked iteratively.  If there's a loop that executes with
"many iterations", but has debug logging that is doing string
concatenation, that would be a good scenario to use the enabled check,
though parameterization with literal values helps in that case. If the
value you are using in a logging call comes from a getter which retrieves
something that's in a literal form, like a getter that returns a boolean
variable, that would be inexpensive.  Calling some list.size() methods are
also pretty cheap.  Even "cheap" operations can add up if the logging is
done iteratively.

Adding a lot of isDebugEnabled checks can make the code less readable and
add many extra lines of code.  Those checks might be necessary to prevent a
decrease in performance, though.  You can always run a profiler to see how
much time you can save when a particular logging level is not enabled, and
update the code with enabled checks as necessary.

On Thu, Sep 20, 2018 at 3:37 PM Mohammed Nadeem 
wrote:

> Hi,
>
> To answer to your question, Nifi has a context logger which is mapped to
> logback.xml file. Generally INFO level of log is defaulted in
> nifi-app.logs.
> In your code, you can simple use getLogger().debug("your message") or INFO,
> ERROR,WARN etc to enable the logging into nifi-app.log with component-id
> etc.
>
> As I mentioned, by default INFO level would go to nifi-app.logs, you can
> change the level in logback.xml under root.
>
> Thanks & Regards,
> Nadeem
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>

[DISCUSS] Closing in on a release of NiFi 1.8.0?

[Jeff]

It looks like we're getting close to a point where we could release NiFi
1.8.0The release tracking page for version 1.8.0 [1] shows 3 "in progress"
and 9 "to do" issues. In addition to what has been tagged with a fix
version of 1.8.0, it looks like NIFI-5516 and NIFI-5585 are close to
completion.

Are there other JIRAs that the community considers necessary for the
release that are close to being resolved, with the goal of getting a
release candidate out in the next couple of weeks?

I'm happy to perform the release manager duties!

[1] https://issues.apache.org/jira/projects/NIFI/versions/12343482

Re: [DISCUSS] Closing in on a release of NiFi 1.8.0?

[Jeff]

Hello everyone!  Next week is probably a good timeframe to aim for a
release candidate, with two major feature PRs recently merged to master:

   - NIFI-5516 <https://issues.apache.org/jira/browse/NIFI-5516> - Allow
   data in a Connection to be Load-Balanced across cluster
   - NIFI-5585 <https://issues.apache.org/jira/browse/NIFI-5585> - Prepare
   Nodes to be Offloaded


To recap, here's a list of other JIRAs mentioned in this thread:

   - NIFI-5402 <https://issues.apache.org/jira/browse/NIFI-5402> - Reduce
   artifact size by only building .zip archive
   - NIFI-5462 <https://issues.apache.org/jira/browse/NIFI-5462> - Refactor
   TLS Toolkit
   - NIFI-5485 <https://issues.apache.org/jira/browse/NIFI-5485> - Enable
   TLS Toolkit (client/server) to sign certificates with external CA
   certificate
   - NIFI-5537 <https://issues.apache.org/jira/browse/NIFI-5537> - Create
   Neo4J cypher execution processor
   - PR 2956 <https://github.com/apache/nifi/pull/2956>
  - Mike Thomsen, this was the specific JIRA to which you were
  referring, right?
   - NIFI-5582 <https://issues.apache.org/jira/browse/NIFI-5582> - Integrate
   legacy behavior of HashAttribute into CryptographicHashAttribute


These JIRAs are marked with a fix version of 1.8.0 that are not currently
resolved:

   - NIFI-4811 <https://issues.apache.org/jira/browse/NIFI-4811> - Use a
   newer version of spring-data-redis
   - PR 2856 <https://github.com/apache/nifi/pull/2856>
   - NIFI-5426 <https://issues.apache.org/jira/browse/NIFI-5426> - Use
   NIO.2 API for ListFile to avoid multiple disk reads
  - PR 2889 <https://github.com/apache/nifi/pull/2889>
   - NIFI-5448 <https://issues.apache.org/jira/browse/NIFI-5448> - Failed
   EL date parsing live-locks processors without a failure relationship
   - NIFI-5665 <https://issues.apache.org/jira/browse/NIFI-5665> - Upgrade
   io.netty dependencies
   - NIFI-5686 <https://issues.apache.org/jira/browse/NIFI-5686> - Test
   failure in TestStandardProcessScheduler
   - PR 3062 <https://github.com/apache/nifi/pull/3062>


On Thu, Oct 4, 2018 at 6:39 AM Mike Thomsen  wrote:

> That's a fair point. Only thing I could add there is that I think we should
> consider a targeted burn down on the PR list as part of 1.9. There are a
> lot of PRs from the last several months that would be good candidates to
> see if we can close them out like MarkLogic and Pulsar.
>
> On Wed, Oct 3, 2018 at 10:34 PM Joe Witt  wrote:
>
> > Mike,
> >
> > Processors in particularly are among the toughest at this point.  We
> > have very very little headroom on dependency size for the full build
> > size that we upload to ASF infra and mirrors.  That and the license
> > review work involved in each...
> >
> > We should really create a way to publish processors on more frequent,
> > irregular intervals where the release work and size/etc.. are far less
> > problematic.  We have another discuss thread on that so I'll leave it
> > there for discussion.  I do share your view that this processor (among
> > several others outstanding) would be really useful but i am definitely
> > thinking we should keep release pace up.  Release more often...release
> > processors separately, etc..
> >
> > Thanks
> > Joe
> > On Wed, Oct 3, 2018 at 9:30 PM Mike Thomsen 
> > wrote:
> > >
> > > I would like to see the Neo4J work that mans2singh is doing get
> included.
> > > Being able to at least partially support a popular graph database would
> > be
> > > a nice feather in our cap.
> > >
> > > On Wed, Oct 3, 2018 at 5:12 PM Andy LoPresto 
> > wrote:
> > >
> > > > I am currently working on a TLS Toolkit refactor (NIFI-5462 &
> > NIFI-5485)
> > > > and HashAttribute updates (NIFI-5582). I believe there are a couple
> > upgrade
> > > > PRs open, and I would really like to see NIFI-5402 (no .tar.gz in the
> > > > build) tackled for this release as well.
> > > >
> > > >
> > > > Andy LoPresto
> > > > alopre...@apache.org
> > > > *alopresto.apa...@gmail.com *
> > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> > > >
> > > > On Oct 3, 2018, at 11:16 AM, Joe Witt  wrote:
> > > >
> > > > Jeff - thanks again for volunteering.  I just went through the open
> > > > items tagged to 1.8.0 to try and shake some loose, close down ones
> > > > that appear to be done but forgotten, and initiate resolution on one
> > > > that is in a dangling state.
> > > >
> > > > Another very 

Re: NIFI single node in cluster mode

[Jeff]

Milan,

If you haven't already done so, please take a look at the NiFi Admin
Guide's sections "Securing Zookeeper" [1] and "Kerberizing NiFi’s ZooKeeper
Client" [2], which should help you configure NiFi to use a kerberized
ZooKeeper.

[1]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#securing_zookeeper
[2]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#zk_kerberos_client

On Sat, Oct 13, 2018 at 9:38 AM Milan Das  wrote:

> Problem is I am using Kerbrized zookeeper and it is failing to create nifi
> basepath. Even if TGT is getting created Authentication is failing.
>
>
> 2018-10-13 13:33:53,573 INFO [Thread-12] org.apache.zookeeper.Login TGT
> refresh thread started.
> 2018-10-13 13:33:53,576 INFO [Thread-12] org.apache.zookeeper.Login TGT
> valid starting at:Sat Oct 13 13:33:53 UTC 2018
> 2018-10-13 13:33:53,576 INFO [Thread-12] org.apache.zookeeper.Login TGT
> expires:  Sun Oct 14 13:33:53 UTC 2018
> 2018-10-13 13:33:53,577 INFO [Thread-12] org.apache.zookeeper.Login TGT
> refresh sleeping until: Sun Oct 14 09:38:53 UTC 2018
> 2018-10-13 13:33:53,577 INFO
> [main-SendThread(ip-172-30-1-132.ec2.internal:2181)]
> o.a.zookeeper.client.ZooKeeperSaslClient Client will use GSSAPI as SASL
> mechanism.
> 2018-10-13 13:33:53,606 INFO [main-EventThread]
> o.a.c.f.state.ConnectionStateManager State change: CONNECTED
> 2018-10-13 13:33:53,616 ERROR
> [main-SendThread(ip-172-30-1-132.ec2.internal:2181)]
> o.a.zookeeper.client.ZooKeeperSaslClient SASL authentication failed using
> login context 'Client'.
> 2018-10-13 13:33:53,723 WARN [main]
> o.a.n.c.l.e.CuratorLeaderElectionManager Unable to determine the Elected
> Leader for role 'Cluster Coordinator' due to
> org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode =
> AuthFailed for /nifi/leaders/Cluster Coordinator; assuming no leader has
> been elected
> 2018-10-13 13:33:53,724 INFO [Curator-Framework-0]
> o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
> 2018-10-13 13:33:53,726 INFO [main]
> o.apache.nifi.controller.FlowController It appears that no Cluster
> Coordinator has been Elected yet. Registering for Cluster Coordinator Role.
>
>
> Thanks,
> Milan Das
>
>
> On 10/12/18, 6:26 PM, "Bryan Bende"  wrote:
>
> There is also another property for the # of candidates to wait for when
> voting, if it sees the # of candidates first it will short circuit the
> time
> period. So setting the candidates to 1 for a single node cluster should
> start immediately.
>
> On Fri, Oct 12, 2018 at 5:59 PM Jon Logan  wrote:
>
> > It waits for election for a specific period of time, which if I
> recall is
> > fairly high (I think 5 minutes?). If you lower this it'll still wait
> for an
> > election but will complete faster (we do 30 seconds, but you could do
> > lower). There's a property controlling this.
> >
> > On Fri, Oct 12, 2018 at 5:41 PM Milan Das  wrote:
> >
> > > Hello Nifi team,
> > >
> > > Is it possible to run a single NIFI node in cluster mode ? I have
> this
> > > requirement because we will add other nodes soon down line.
> > >
> > > I tried that by setting  “nifi.cluster.is.node” but and zookeeper
> > setting.
> > > But seems it waits ever for election.
> > >
> > >
> > >
> > > Appreciate your thoughts.
> > >
> > >
> > >
> > > Thanks,
> > >
> > > Milan Das
> > >
> > >
> >
> --
> Sent from Gmail Mobile
>
>
>
>

Re: [DISCUSS] Closing in on a release of NiFi 1.8.0?

[Jeff]

Sivaprasanna,

Thanks for submitting a pull request for that issue!  Later today or
tomorrow I'll have to check to see if I've already used up my free-tier
access to Azure.  If I still have access, I can review your PR and we'll
get it into 1.8.0.

On Sun, Oct 14, 2018 at 4:30 AM Sivaprasanna 
wrote:

> All - Just found one bug with DeleteAzureBlobStorage processor. It was
> shared by one user on StackOverflow [1] and I later confirmed it. It looks
> to be introduced by NIFI-4199. I have created a Jira [2] and made the
> necessary changes (not huge, just few lines) and raised a PR [3]. I think,
> if we can spend a little time in getting it reviewed, we can mark it for
> 1.8.0. Thoughts?
>
> [1] -
>
> https://stackoverflow.com/questions/52766991/apache-nifi-deleteazureblobstorage-processor-is-throwing-an-error
> [2] - https://issues.apache.org/jira/browse/NIFI-5698
> [3] - https://github.com/apache/nifi/pull/3073
>
> -
> Sivaprasanna
>
> On Fri, Oct 12, 2018 at 9:05 PM Mike Thomsen 
> wrote:
>
> > 4811 should be ready for review now. Rebased and cleaned it up with a
> full
> > listing of the Spring dependencies.
> >
> > On Fri, Oct 12, 2018 at 11:23 AM Joe Witt  wrote:
> >
> > > Jeff,
> > >
> > > I think for anything not tagged to 1.8.0 we just keep rolling.  For
> > > anything tagged 1.8.0 that should not be we should remove it until
> > > ready.  For things tagged to 1.8.0 that cannot be moved we should
> > > resolve.  For the tagged 1.8.0 section you had.
> > >
> > >- NIFI-4811 <https://issues.apache.org/jira/browse/NIFI-4811> -
> Use a
> > >newer version of spring-data-redis
> > >- PR 2856 <https://github.com/apache/nifi/pull/2856>
> > > *This needs to be resolved by either reverting the commit or ensuring
> > > L&N accurately reflects all.  We have to do this always and for every
> > > nar.  The process isnt easy or fun but it is necessary to produce
> > > valid ASF releases.  Landing commits which change dependencies
> > > requires this due diligence.  Now, we've put a lot of energy into
> > > updating Spring dependencies because some older Spring libs had
> > > vulnerabilities which while we likely aren't exposed to them we want
> > > to fix in due course.  So reverting may require more analysis than if
> > > we were just get L&N fixed with this new change.  I commented on the
> > > JIRA.  But this needs to be resolved.
> > >
> > >
> > >- NIFI-5426 <https://issues.apache.org/jira/browse/NIFI-5426> - Use
> > >NIO.2 API for ListFile to avoid multiple disk reads
> > >   - PR 2889 <https://github.com/apache/nifi/pull/2889>
> > > *This just needed to be marked resolved.  The commit went in the day
> > > after we cut 1.7.1.  So this one is sorted.
> > >
> > >- NIFI-5448 <https://issues.apache.org/jira/browse/NIFI-5448> -
> > Failed
> > >EL date parsing live-locks processors without a failure relationship
> > > * The commit needs to be reverted.  I'm working on that now.  Once the
> > > discsusion/concerns are addressed this can get dealt with.
> > >
> > >- NIFI-5665 <https://issues.apache.org/jira/browse/NIFI-5665> -
> > Upgrade
> > >io.netty dependencies
> > > * This looks important to get resolved if possible as old netty libs
> > > are on the list of things with vulnerabilities.
> > >
> > >- NIFI-5686 <https://issues.apache.org/jira/browse/NIFI-5686> -
> Test
> > >failure in TestStandardProcessScheduler
> > >- PR 3062 <https://github.com/apache/nifi/pull/3062>
> > > * This has a PR but a test, possibly two, failed in one of the travis
> > > runs and it is clearly related.  I ignored one of those tests in a
> > > previous run.  We must deal with brittle tests.  But the underlying
> > > problem is important to solve here so either the tests needs improved
> > > or we still have an issue.  Not clear but worth some focus.
> > >
> > > note: I intend to reference updates to libraries that have known
> > > vulnerabilities and do so in a far less subtle manner than we had.  We
> > > aren't acknowledging that NiFi is or exposes vulnerabilities but we
> > > are and should be clear when we're updating dependencies that do have
> > > them (even if we're not exposed to them) so that some of these commits
> > > aren't so mysterious.  It creates far more confusion than is worth.
> &g

Re: [DISCUSS] Closing in on a release of NiFi 1.8.0?

[Jeff]

NiFi Devs,

The Release page [1] for 1.8.0 now reports that all issues are done!  I'd
like to start the release candidate preparation tomorrow, around 1200 EST.

Thanks to everyone for all the great work that's been done!  196 issues
resolved in this version with some great new features!

[1] https://issues.apache.org/jira/projects/NIFI/versions/12343482

On Mon, Oct 15, 2018 at 7:30 AM Sivaprasanna 
wrote:

> Great. Thanks. :)
>
> -
> Sivaprasanna
>
> On Mon, Oct 15, 2018 at 7:09 AM Koji Kawamura 
> wrote:
>
> > Jeff, Sivasprasanna,
> >
> > NIFI-5698 (PR3073) Fixing DeleteAzureBlob bug is merged.
> >
> > Thanks,
> > Koji
> > On Mon, Oct 15, 2018 at 10:18 AM Koji Kawamura 
> > wrote:
> > >
> > > Thank you for the fix Sivaprasanna,
> > > I have Azure account. Reviewing it now.
> > >
> > > Koji
> > > On Sun, Oct 14, 2018 at 11:21 PM Jeff  wrote:
> > > >
> > > > Sivaprasanna,
> > > >
> > > > Thanks for submitting a pull request for that issue!  Later today or
> > > > tomorrow I'll have to check to see if I've already used up my
> free-tier
> > > > access to Azure.  If I still have access, I can review your PR and
> > we'll
> > > > get it into 1.8.0.
> > > >
> > > > On Sun, Oct 14, 2018 at 4:30 AM Sivaprasanna <
> > sivaprasanna...@gmail.com>
> > > > wrote:
> > > >
> > > > > All - Just found one bug with DeleteAzureBlobStorage processor. It
> > was
> > > > > shared by one user on StackOverflow [1] and I later confirmed it.
> It
> > looks
> > > > > to be introduced by NIFI-4199. I have created a Jira [2] and made
> the
> > > > > necessary changes (not huge, just few lines) and raised a PR [3]. I
> > think,
> > > > > if we can spend a little time in getting it reviewed, we can mark
> it
> > for
> > > > > 1.8.0. Thoughts?
> > > > >
> > > > > [1] -
> > > > >
> > > > >
> >
> https://stackoverflow.com/questions/52766991/apache-nifi-deleteazureblobstorage-processor-is-throwing-an-error
> > > > > [2] - https://issues.apache.org/jira/browse/NIFI-5698
> > > > > [3] - https://github.com/apache/nifi/pull/3073
> > > > >
> > > > > -
> > > > > Sivaprasanna
> > > > >
> > > > > On Fri, Oct 12, 2018 at 9:05 PM Mike Thomsen <
> mikerthom...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > 4811 should be ready for review now. Rebased and cleaned it up
> > with a
> > > > > full
> > > > > > listing of the Spring dependencies.
> > > > > >
> > > > > > On Fri, Oct 12, 2018 at 11:23 AM Joe Witt 
> > wrote:
> > > > > >
> > > > > > > Jeff,
> > > > > > >
> > > > > > > I think for anything not tagged to 1.8.0 we just keep rolling.
> > For
> > > > > > > anything tagged 1.8.0 that should not be we should remove it
> > until
> > > > > > > ready.  For things tagged to 1.8.0 that cannot be moved we
> should
> > > > > > > resolve.  For the tagged 1.8.0 section you had.
> > > > > > >
> > > > > > >- NIFI-4811 <
> https://issues.apache.org/jira/browse/NIFI-4811>
> > -
> > > > > Use a
> > > > > > >newer version of spring-data-redis
> > > > > > >- PR 2856 <https://github.com/apache/nifi/pull/2856>
> > > > > > > *This needs to be resolved by either reverting the commit or
> > ensuring
> > > > > > > L&N accurately reflects all.  We have to do this always and for
> > every
> > > > > > > nar.  The process isnt easy or fun but it is necessary to
> produce
> > > > > > > valid ASF releases.  Landing commits which change dependencies
> > > > > > > requires this due diligence.  Now, we've put a lot of energy
> into
> > > > > > > updating Spring dependencies because some older Spring libs had
> > > > > > > vulnerabilities which while we likely aren't exposed to them we
> > want
> > > > > > > to fix in due course.  So reverting may require more analysis
> > than if
> > > > > > > we were just 

Re: [DISCUSS] Closing in on a release of NiFi 1.8.0?

[Jeff]

Since NIFI-5562 is an improvement JIRA, and we're cutting it very close to
the release candidate, it may not make it into the release.  However, since
Mark Payne is investigating some issues that may have been introduced in
1.8.0, I can delay creating RC1.

On Tue, Oct 16, 2018 at 10:24 AM Mark Payne  wrote:

> Jeff / all,
>
> I ran into an issue with NIFI-375 and re-opened the ticket. If a processor
> is stopped or started in a cluster,
> the stats that come back in the response are incorrect because the
> response is not being properly merged
> from all nodes in the cluster.
>
> I also have run into a couple of other issues that appear to have been
> introduced in 1.8. I have not yet created JIRA's
> for them because I want to understand the issues better before trying to
> write it up. I am seeing, for example, some errors
> in the logs indicating that provenance events don't have a FlowFile UUID
> assigned to them when being read from the
> repository. This seems to have been introduced in 1.8 so would like to get
> a resolution before releasing.
>
> Thanks
> -Mark
>
>
> > On Oct 15, 2018, at 6:33 PM, Jeff  wrote:
> >
> > NiFi Devs,
> >
> > The Release page [1] for 1.8.0 now reports that all issues are done!  I'd
> > like to start the release candidate preparation tomorrow, around 1200
> EST.
> >
> > Thanks to everyone for all the great work that's been done!  196 issues
> > resolved in this version with some great new features!
> >
> > [1] https://issues.apache.org/jira/projects/NIFI/versions/12343482
> >
> > On Mon, Oct 15, 2018 at 7:30 AM Sivaprasanna 
> > wrote:
> >
> >> Great. Thanks. :)
> >>
> >> -
> >> Sivaprasanna
> >>
> >> On Mon, Oct 15, 2018 at 7:09 AM Koji Kawamura 
> >> wrote:
> >>
> >>> Jeff, Sivasprasanna,
> >>>
> >>> NIFI-5698 (PR3073) Fixing DeleteAzureBlob bug is merged.
> >>>
> >>> Thanks,
> >>> Koji
> >>> On Mon, Oct 15, 2018 at 10:18 AM Koji Kawamura  >
> >>> wrote:
> >>>>
> >>>> Thank you for the fix Sivaprasanna,
> >>>> I have Azure account. Reviewing it now.
> >>>>
> >>>> Koji
> >>>> On Sun, Oct 14, 2018 at 11:21 PM Jeff  wrote:
> >>>>>
> >>>>> Sivaprasanna,
> >>>>>
> >>>>> Thanks for submitting a pull request for that issue!  Later today or
> >>>>> tomorrow I'll have to check to see if I've already used up my
> >> free-tier
> >>>>> access to Azure.  If I still have access, I can review your PR and
> >>> we'll
> >>>>> get it into 1.8.0.
> >>>>>
> >>>>> On Sun, Oct 14, 2018 at 4:30 AM Sivaprasanna <
> >>> sivaprasanna...@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> All - Just found one bug with DeleteAzureBlobStorage processor. It
> >>> was
> >>>>>> shared by one user on StackOverflow [1] and I later confirmed it.
> >> It
> >>> looks
> >>>>>> to be introduced by NIFI-4199. I have created a Jira [2] and made
> >> the
> >>>>>> necessary changes (not huge, just few lines) and raised a PR [3]. I
> >>> think,
> >>>>>> if we can spend a little time in getting it reviewed, we can mark
> >> it
> >>> for
> >>>>>> 1.8.0. Thoughts?
> >>>>>>
> >>>>>> [1] -
> >>>>>>
> >>>>>>
> >>>
> >>
> https://stackoverflow.com/questions/52766991/apache-nifi-deleteazureblobstorage-processor-is-throwing-an-error
> >>>>>> [2] - https://issues.apache.org/jira/browse/NIFI-5698
> >>>>>> [3] - https://github.com/apache/nifi/pull/3073
> >>>>>>
> >>>>>> -
> >>>>>> Sivaprasanna
> >>>>>>
> >>>>>> On Fri, Oct 12, 2018 at 9:05 PM Mike Thomsen <
> >> mikerthom...@gmail.com
> >>>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> 4811 should be ready for review now. Rebased and cleaned it up
> >>> with a
> >>>>>> full
> >>>>>>> listing of the Spring dependencies.
> >>>>>>>
> >>>>>>> On Fri, Oct 12, 2018 at 

Re: Fixing unstable nifi cluster.

[Jeff]

Hello,

Pierre's suggestions should be helpful for you, since you are running
Zookeeper on the same nodes as NiFi.  If it's possible for you to run
Zookeeper on separate hosts from NiFi so that ZK and NiFi are not
colocated, you should see better results.

- Jeff

On Tue, Oct 16, 2018 at 8:03 AM Pierre Villard 
wrote:

> Hi,
>
> Can you try increasing the below parameters? That's usually what I
> recommend, our default values being probably a bit too aggressively low.
>
> nifi.zookeeper.connect.timeout=15 secs
> nifi.zookeeper.session.timeout=15 secs
> nifi.cluster.node.read.timeout=30 sec
>
> Pierre
>
> Le mar. 16 oct. 2018 à 13:02, ashwin konale  a
> écrit :
>
> > Hi,
> > We have a 3 node nifi cluster (With separate zookeper instances running
> in
> > the same machines) which pulls the data from mysql and write to hdfs. I
> am
> > frequently running into problems with cluster. Nodes keeps disconnecting
> > from each other, primary nodes keeps switching and sometimes it just goes
> > into zombie state when I just cannot access the ui. I have followed best
> > practices guide and tweaked params in nifi.properties, have switched
> > provenanceRepositoryImplementation to volatile because cluster was not
> able
> > to keep up with incoming traffic. Data traffic is not high at all
> (4Mbps).
> > This is the message I frequently get from the logs.
> >
> > *INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State
> change:
> > LOST*
> > *INFO [Curator-ConnectionStateManager-0]
> > o.a.n.c.l.e.CuratorLeaderElectionManager
> >
> >
> org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@56ebedec
> > Connection State changed to LOST*
> > *INFO [Curator-ConnectionStateManager-0]
> > o.a.n.c.l.e.CuratorLeaderElectionManager
> >
> >
> org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@1b0e2055
> > Connection State changed to LOST*
> > *INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State
> change:
> > RECONNECTED*
> >
> > Am I doing something wrong with cluster setup ? Can someone give me some
> > guidance on how to go about debugging this issue ? What kind of system
> > metrics to look at etc.
> >
> > Ashwin
> >
>

Re: [DISCUSS] Closing in on a release of NiFi 1.8.0?

[Jeff]

A quick update on RC1 for NiFi 1.8.0.  There are some L&N issues that are
currently being worked, so I'd like to hold off on creating the RC until
they are resolved.  Sorry for the delay!

On Tue, Oct 16, 2018 at 3:46 PM Nathan Gough  wrote:

> So, taking a closer look at this PR and doing some further testing I've
> found that upgrading Guava is going to require some more careful
> assessment, as the changes will affect core NiFi functionality around
> clustering. I will hold off on this change until after the release to have
> more time to upgrade and test.
>
> Nathan
>
> On 10/16/18, 10:58 AM, "Nathan Gough"  wrote:
>
> Hi Mike,
>
> Sure I can look at fixing up PR-2977 today.
>
> Nathan
>
> On 10/16/18, 6:13 AM, "Mike Thomsen"  wrote:
>
> Does 5562 need to be addressed in 1.8?
>
> https://github.com/apache/nifi/pull/2977
>
> On Mon, Oct 15, 2018 at 6:33 PM Jeff  wrote:
>
> > NiFi Devs,
> >
> > The Release page [1] for 1.8.0 now reports that all issues are
> done!  I'd
> > like to start the release candidate preparation tomorrow, around
> 1200 EST.
> >
> > Thanks to everyone for all the great work that's been done!  196
> issues
> > resolved in this version with some great new features!
> >
> > [1]
> https://issues.apache.org/jira/projects/NIFI/versions/12343482
> >
> > On Mon, Oct 15, 2018 at 7:30 AM Sivaprasanna <
> sivaprasanna...@gmail.com>
> > wrote:
> >
> > > Great. Thanks. :)
> > >
> > > -
> > > Sivaprasanna
> > >
> > > On Mon, Oct 15, 2018 at 7:09 AM Koji Kawamura <
> ijokaruma...@gmail.com>
> > > wrote:
> > >
> > > > Jeff, Sivasprasanna,
> > > >
> > > > NIFI-5698 (PR3073) Fixing DeleteAzureBlob bug is merged.
> > > >
> > > > Thanks,
> > > > Koji
> > > > On Mon, Oct 15, 2018 at 10:18 AM Koji Kawamura <
> ijokaruma...@gmail.com
> > >
> > > > wrote:
> > > > >
> > > > > Thank you for the fix Sivaprasanna,
> > > > > I have Azure account. Reviewing it now.
> > > > >
> > > > > Koji
> > > > > On Sun, Oct 14, 2018 at 11:21 PM Jeff 
> wrote:
> > > > > >
> > > > > > Sivaprasanna,
> > > > > >
> > > > > > Thanks for submitting a pull request for that issue!
> Later today
> > or
> > > > > > tomorrow I'll have to check to see if I've already used
> up my
> > > free-tier
> > > > > > access to Azure.  If I still have access, I can review
> your PR and
> > > > we'll
> > > > > > get it into 1.8.0.
> > > > > >
> > > > > > On Sun, Oct 14, 2018 at 4:30 AM Sivaprasanna <
> > > > sivaprasanna...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > All - Just found one bug with DeleteAzureBlobStorage
> processor.
> > It
> > > > was
> > > > > > > shared by one user on StackOverflow [1] and I later
> confirmed it.
> > > It
> > > > looks
> > > > > > > to be introduced by NIFI-4199. I have created a Jira
> [2] and made
> > > the
> > > > > > > necessary changes (not huge, just few lines) and
> raised a PR
> > [3]. I
> > > > think,
> > > > > > > if we can spend a little time in getting it reviewed,
> we can mark
> > > it
> > > > for
> > > > > > > 1.8.0. Thoughts?
> > > > > > >
> > > > > > > [1] -
> > > > > > >
> > > > > > >
> > > >
> > >
> >
> https://stackoverflow.com/questions/52766991/apache-nifi-deleteazureblobstorage-processor-is-throwing-an-error
> > > > > > > [2] - htt

Re: [DISCUSS] Closing in on a release of NiFi 1.8.0?

[Jeff]

The L&N issues previously mentioned have been resolved, and I'm starting
the release prep for RC1 now.

On Tue, Oct 16, 2018 at 4:52 PM Jeff  wrote:

> A quick update on RC1 for NiFi 1.8.0.  There are some L&N issues that are
> currently being worked, so I'd like to hold off on creating the RC until
> they are resolved.  Sorry for the delay!
>
> On Tue, Oct 16, 2018 at 3:46 PM Nathan Gough  wrote:
>
>> So, taking a closer look at this PR and doing some further testing I've
>> found that upgrading Guava is going to require some more careful
>> assessment, as the changes will affect core NiFi functionality around
>> clustering. I will hold off on this change until after the release to have
>> more time to upgrade and test.
>>
>> Nathan
>>
>> On 10/16/18, 10:58 AM, "Nathan Gough"  wrote:
>>
>> Hi Mike,
>>
>> Sure I can look at fixing up PR-2977 today.
>>
>> Nathan
>>
>> On 10/16/18, 6:13 AM, "Mike Thomsen"  wrote:
>>
>> Does 5562 need to be addressed in 1.8?
>>
>> https://github.com/apache/nifi/pull/2977
>>
>> On Mon, Oct 15, 2018 at 6:33 PM Jeff  wrote:
>>
>> > NiFi Devs,
>> >
>> > The Release page [1] for 1.8.0 now reports that all issues are
>> done!  I'd
>> > like to start the release candidate preparation tomorrow,
>> around 1200 EST.
>> >
>> > Thanks to everyone for all the great work that's been done!
>> 196 issues
>> > resolved in this version with some great new features!
>> >
>> > [1]
>> https://issues.apache.org/jira/projects/NIFI/versions/12343482
>> >
>> > On Mon, Oct 15, 2018 at 7:30 AM Sivaprasanna <
>> sivaprasanna...@gmail.com>
>> > wrote:
>> >
>> > > Great. Thanks. :)
>> > >
>> > > -
>> > > Sivaprasanna
>> > >
>> > > On Mon, Oct 15, 2018 at 7:09 AM Koji Kawamura <
>> ijokaruma...@gmail.com>
>> > > wrote:
>> > >
>> > > > Jeff, Sivasprasanna,
>> > > >
>> > > > NIFI-5698 (PR3073) Fixing DeleteAzureBlob bug is merged.
>> > > >
>> > > > Thanks,
>> > > > Koji
>> > > > On Mon, Oct 15, 2018 at 10:18 AM Koji Kawamura <
>> ijokaruma...@gmail.com
>> > >
>> > > > wrote:
>> > > > >
>> > > > > Thank you for the fix Sivaprasanna,
>> > > > > I have Azure account. Reviewing it now.
>> > > > >
>> > > > > Koji
>> > > > > On Sun, Oct 14, 2018 at 11:21 PM Jeff 
>> wrote:
>> > > > > >
>> > > > > > Sivaprasanna,
>> > > > > >
>> > > > > > Thanks for submitting a pull request for that issue!
>> Later today
>> > or
>> > > > > > tomorrow I'll have to check to see if I've already used
>> up my
>> > > free-tier
>> > > > > > access to Azure.  If I still have access, I can review
>> your PR and
>> > > > we'll
>> > > > > > get it into 1.8.0.
>> > > > > >
>> > > > > > On Sun, Oct 14, 2018 at 4:30 AM Sivaprasanna <
>> > > > sivaprasanna...@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > All - Just found one bug with DeleteAzureBlobStorage
>> processor.
>> > It
>> > > > was
>> > > > > > > shared by one user on StackOverflow [1] and I later
>> confirmed it.
>> > > It
>> > > > looks
>> > > > > > > to be introduced by NIFI-4199. I have created a Jira
>> [2] and made
>> > > the
>> > > > > > > necessary changes (not huge, just few lines) and
>> raised a PR
>> > [3]. I
>> > > > think,
>> > > > > > > if we can spend a little

[VOTE] Release Apache NiFi 1.8.0

[Jeff]

Hello,

I am pleased to be calling this vote for the source release of Apache NiFi
nifi-1.8.0.

The source zip, including signatures, digests, etc. can be found at:
https://repository.apache.org/content/repositories/orgapachenifi-1133

The Git tag is nifi-1.8.0-RC1
The Git commit ID is 9b02d58626ca874ed2ed3e0bbe530512cfa0dbf8
https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=9b02d58626ca874ed2ed3e0bbe530512cfa0dbf8

Checksums of nifi-1.8.0-source-release.zip:
SHA256: 3ec90a7f153e507d7bba2400d6dafac02641d6f7afc7a954fed959191073ce21
SHA512:
8b9d944da1833bfb645f502107cab98a555e3b2a7602c5ff438407272c86defdeebe18625c5ad9dfb3f344397314569e97220a35f2438182a79a700caa90721e

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/jstorck.asc

KEYS file available here:
https://dist.apache.org/repos/dist/release/nifi/KEYS

204 issues were closed/resolved for this release:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12343482

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.8.0

The vote will be open for 72 hours.
Please download the release candidate and evaluate the necessary items
including checking hashes, signatures, build
from source, and test. Then please vote:

[ ] +1 Release this package as nifi-1.8.0
[ ] +0 no opinion
[ ] -1 Do not release this package because...

Apache NiFi 1.8.0 RC1 Release Helper Guide

[Jeff]

Hello Apache NiFi community,

Please find the associated guidance to help those interested in
validating/verifying the release so they can vote.

# Download latest KEYS file:
https://dist.apache.org/repos/dist/dev/nifi/KEYS

# Import keys file:
gpg --import KEYS

# [optional] Clear out local maven artifact repository

# Pull down nifi-1.8.0 source release artifacts for review:

wget
https://dist.apache.org/repos/dist/dev/nifi/nifi-1.8.0/nifi-1.8.0-source-release.zip
wget
https://dist.apache.org/repos/dist/dev/nifi/nifi-1.8.0/nifi-1.8.0-source-release.zip.asc
wget
https://dist.apache.org/repos/dist/dev/nifi/nifi-1.8.0/nifi-1.8.0-source-release.zip.sha256
wget
https://dist.apache.org/repos/dist/dev/nifi/nifi-1.8.0/nifi-1.8.0-source-release.zip.sha512

# Verify the signature
gpg --verify nifi-1.8.0-source-release.zip.asc

# Verify the hashes (sha256, sha512) match the source and what was provided
in the vote email thread
shasum -a 256 nifi-1.8.0-source-release.zip
shasum -a 512 nifi-1.8.0-source-release.zip

# Unzip nifi-1.8.0-source-release.zip

# Verify the build works including release audit tool (RAT) checks
cd nifi-1.8.0
mvn clean install -Pcontrib-check,include-grpc

# Verify the contents contain a good README, NOTICE, and LICENSE.

# Verify the git commit ID is correct

# Verify the RC was branched off the correct git commit ID

# Look at the resulting convenience binary as found in nifi-assembly/target

# Make sure the README, NOTICE, and LICENSE are present and correct

# Run the resulting convenience binary and make sure it works as expected

# Send a response to the vote thread indicating a +1, 0, -1 based on your
findings.

Thank you for your time and effort to validate the release!

Re: Apache NiFi 1.8.0 RC1 Release Helper Guide

[Jeff]

Sorry Otto!  The tag [1] should be there now.

[1] 
*https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=tag;h=2fdc819e65481e609532312d18a41d30d58f16f3
<https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=tag;h=2fdc819e65481e609532312d18a41d30d58f16f3>*

On Thu, Oct 18, 2018 at 8:34 AM Otto Fowler  wrote:

> I don’t see a tag for nifi-1.8.0-RC1
>
>
> On October 18, 2018 at 00:01:06, Jeff (jsto...@apache.org) wrote:
>
> Hello Apache NiFi community,
>
> Please find the associated guidance to help those interested in
> validating/verifying the release so they can vote.
>
> # Download latest KEYS file:
> https://dist.apache.org/repos/dist/dev/nifi/KEYS
>
> # Import keys file:
> gpg --import KEYS
>
> # [optional] Clear out local maven artifact repository
>
> # Pull down nifi-1.8.0 source release artifacts for review:
>
> wget
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.8.0/nifi-1.8.0-source-release.zip
> wget
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.8.0/nifi-1.8.0-source-release.zip.asc
> wget
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.8.0/nifi-1.8.0-source-release.zip.sha256
> wget
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.8.0/nifi-1.8.0-source-release.zip.sha512
>
> # Verify the signature
> gpg --verify nifi-1.8.0-source-release.zip.asc
>
> # Verify the hashes (sha256, sha512) match the source and what was provided
> in the vote email thread
> shasum -a 256 nifi-1.8.0-source-release.zip
> shasum -a 512 nifi-1.8.0-source-release.zip
>
> # Unzip nifi-1.8.0-source-release.zip
>
> # Verify the build works including release audit tool (RAT) checks
> cd nifi-1.8.0
> mvn clean install -Pcontrib-check,include-grpc
>
> # Verify the contents contain a good README, NOTICE, and LICENSE.
>
> # Verify the git commit ID is correct
>
> # Verify the RC was branched off the correct git commit ID
>
> # Look at the resulting convenience binary as found in nifi-assembly/target
>
> # Make sure the README, NOTICE, and LICENSE are present and correct
>
> # Run the resulting convenience binary and make sure it works as expected
>
> # Send a response to the vote thread indicating a +1, 0, -1 based on your
> findings.
>
> Thank you for your time and effort to validate the release!
>

[CANCEL] [VOTE] Release Apache NiFi 1.8.0

[Jeff]

Given the issue with the SSLContextService, I'm canceling the vote.

I will create RC2 tomorrow, with a 96-hour voting window since it's the
weekend.

On Fri, Oct 19, 2018 at 4:35 PM Nathan Gough  wrote:

> -1 (revote)
>
> On further testing I have found that the SSLContextService does not work
> as expected due to this ticket
> https://jira.apache.org/jira/browse/NIFI-4558 and the related PR. This
> makes it difficult or impossible to use the SSLContextService as I believe
> the customValidate() method takes in user input and not the default
> settings. I would say this is a blocker for nifi-1.8.0-RC1. I have
> submitted a PR to revert this change:
> https://github.com/apache/nifi/pull/3097.
>
> My apologies,
> Nathan
>
> On 10/19/18, 3:29 PM, "Andrew Lim"  wrote:
>
> +1 (non-binding)
>
> -Ran full clean install on OS X (10.11.6)
> -Tested secure cluster; used UI to: disconnect/connect nodes; offload
> nodes; load balance connections
> -Reviewed documentation
>
> Drew
>
> > On Oct 17, 2018, at 11:59 PM, Jeff  wrote:
> >
> > Hello,
> >
> > I am pleased to be calling this vote for the source release of
> Apache NiFi
> > nifi-1.8.0.
> >
> > The source zip, including signatures, digests, etc. can be found at:
> >
> https://repository.apache.org/content/repositories/orgapachenifi-1133
> >
> > The Git tag is nifi-1.8.0-RC1
> > The Git commit ID is 9b02d58626ca874ed2ed3e0bbe530512cfa0dbf8
> >
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=9b02d58626ca874ed2ed3e0bbe530512cfa0dbf8
> >
> > Checksums of nifi-1.8.0-source-release.zip:
> > SHA256:
> 3ec90a7f153e507d7bba2400d6dafac02641d6f7afc7a954fed959191073ce21
> > SHA512:
> >
> 8b9d944da1833bfb645f502107cab98a555e3b2a7602c5ff438407272c86defdeebe18625c5ad9dfb3f344397314569e97220a35f2438182a79a700caa90721e
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/jstorck.asc
> >
> > KEYS file available here:
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > 204 issues were closed/resolved for this release:
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12343482
> >
> > Release note highlights can be found here:
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.8.0
> >
> > The vote will be open for 72 hours.
> > Please download the release candidate and evaluate the necessary
> items
> > including checking hashes, signatures, build
> > from source, and test. Then please vote:
> >
> > [ ] +1 Release this package as nifi-1.8.0
> > [ ] +0 no opinion
> > [ ] -1 Do not release this package because...
>
>
>
>
>

[VOTE] Release Apache NiFi 1.8.0 (RC2)

[Jeff]

Hello,

I am pleased to be calling this vote for the source release of Apache NiFi
nifi-1.8.0.

The source zip, including signatures, digests, etc. can be found at:
https://repository.apache.org/content/repositories/orgapachenifi-1134

The Git tag is nifi-1.8.0-RC2
The Git commit ID is 19bdd375c32c97e2b7dfd41e5ffe65f5e1eb2435
https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=19bdd375c32c97e2b7dfd41e5ffe65f5e1eb2435

Checksums of nifi-1.8.0-source-release.zip:
SHA256: 72dc2934f70f41e0c62e0aeb2bdc48e9feaa743dc06319cbed42da04bdc0f827
SHA512:
012194f79d4bd5060032588e29f5e9c4240aa5e4758946a6cbcc89c0a1499de9db0c46d3f76e5ee694f0f9345c5f1bee3f3e315ef6fcc1194447958cb3f8b003

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/jstorck.asc

KEYS file available here:
https://dist.apache.org/repos/dist/release/nifi/KEYS

204 issues were closed/resolved for this release:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12343482

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.8.0

The vote will be open for 96 hours.
Please download the release candidate and evaluate the necessary items
including checking hashes, signatures, build
from source, and test. Then please vote:

[ ] +1 Release this package as nifi-1.8.0
[ ] +0 no opinion
[ ] -1 Do not release this package because...

[CANCEL] [VOTE] Release Apache NiFi 1.8.0 (RC2)

[Jeff]

I agree with the move to cancel the vote, and fix this issue before RC3 is
created.

The NiFi 1.8.0 RC2 vote is now canceled, and I'll create RC3 after the fix
has been merged to master.

Thank you to the reviewers that have taken the time to review RC1 and RC2.

On Mon, Oct 22, 2018 at 12:06 PM Mark Payne  wrote:

> -1 (binding)
>
> Unfortunately I ran into a bug that I think warrants sinking this vote. I
> created a JIRA [1] for it.
> It looks like a bug was introduced that can result in a
> processor/port/funnel deadlocking when
> attempting to pull from a Connection or put to a Connection.
>
> Thanks
> -Mark
>
> [1] https://issues.apache.org/jira/browse/NIFI-5736
>
>
> On Oct 20, 2018, at 11:11 PM, Jeff  jsto...@apache.org>> wrote:
>
> Hello,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> nifi-1.8.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1134
>
> The Git tag is nifi-1.8.0-RC2
> The Git commit ID is 19bdd375c32c97e2b7dfd41e5ffe65f5e1eb2435
>
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=19bdd375c32c97e2b7dfd41e5ffe65f5e1eb2435
>
> Checksums of nifi-1.8.0-source-release.zip:
> SHA256: 72dc2934f70f41e0c62e0aeb2bdc48e9feaa743dc06319cbed42da04bdc0f827
> SHA512:
>
> 012194f79d4bd5060032588e29f5e9c4240aa5e4758946a6cbcc89c0a1499de9db0c46d3f76e5ee694f0f9345c5f1bee3f3e315ef6fcc1194447958cb3f8b003
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/jstorck.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 204 issues were closed/resolved for this release:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12343482
>
> Release note highlights can be found here:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.8.0
>
> The vote will be open for 96 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test. Then please vote:
>
> [ ] +1 Release this package as nifi-1.8.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>
>

[VOTE] Release Apache NiFi 1.8.0 (RC3)

[Jeff]

Hello,

I am pleased to be calling this vote for the source release of Apache NiFi
nifi-1.8.0.

The source zip, including signatures, digests, etc. can be found at:
https://repository.apache.org/content/repositories/orgapachenifi-1135

The Git tag is nifi-1.8.0-RC3
The Git commit ID is 98aabf2c50f857efc72fd6f2bfdd9965b97fa195
https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=98aabf2c50f857efc72fd6f2bfdd9965b97fa195

Checksums of nifi-1.8.0-source-release.zip:
SHA256: 6ec21c36ebb232f344493a4aeb5086eed0c462c576e11a79abed8149bc8b65c3
SHA512:
846aecd4eb497a3b7dee7d1911b02453b8162b6c87e39f3df837744a478212e2e3e3615921079d29c2804671f26ecd05b04ce46a4bb69e8911fc185e27be9c24

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/jstorck.asc

KEYS file available here:
https://dist.apache.org/repos/dist/release/nifi/KEYS

209 issues were closed/resolved for this release:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12343482

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.8.0

The vote will be open for 72 hours.
Please download the release candidate and evaluate the necessary items
including checking hashes, signatures, build
from source, and test. Then please vote:

[ ] +1 Release this package as nifi-1.8.0
[ ] +0 no opinion
[ ] -1 Do not release this package because...

[ANNOUNCE] Apache NiFi 1.8.0 release

[Jeff]

Hello

The Apache NiFi team would like to announce the release of Apache NiFi
1.8.0.

Apache NiFi is an easy to use, powerful, and reliable system to process and
distribute
data.  Apache NiFi was made for dataflow.  It supports highly configurable
directed graphs
of data routing, transformation, and system mediation logic.

More details on Apache NiFi can be found here:
https://nifi.apache.org/

The release artifacts can be downloaded from here:
https://nifi.apache.org/download.html

Maven artifacts have been made available here:
https://repository.apache.org/content/repositories/releases/org/apache/nifi/

Issues closed/resolved for this list can be found here:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12343482

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.8.0

Thank you
The Apache NiFi team

Re: [ANNOUNCE] New Apache NiFi Committer Nathan Gough

[Jeff]

Congrats, Nathan!

On Thu, Jan 3, 2019 at 8:02 AM Jeremy Dyer  wrote:

> Congratulations Nathan!
>
> On Thu, Jan 3, 2019 at 6:52 AM Mike Thomsen 
> wrote:
>
> > Congratulations!
> >
> > On Thu, Jan 3, 2019 at 6:51 AM Otto Fowler 
> > wrote:
> >
> > > Congratulations Nathan!
> > >
> > >
> > > On January 2, 2019 at 21:30:42, Tony Kurc (tk...@apache.org) wrote:
> > >
> > > On behalf of the Apache NiFI PMC, I am very pleased to announce that
> > Nathan
> > > has accepted the PMC's invitation to become a committer on the Apache
> > NiFi
> > > project. We greatly appreciate all of Nathan's hard work and generous
> > > contributions to the project. We look forward to his continued
> > involvement
> > > in the project.
> > >
> > > What stood out for the PMC was Nathan's long history of code
> contribution
> > > especially in the area of security, and his always helpful conduct on
> the
> > > mailing lists, the jiras, reviews, and releases. Thanks Nathan!
> > >
> > > Welcome and congratulations!
> > >
> > > - Tony
> > >
> >
>

Re: [ANNOUNCE] New Apache NiFi Committer Ed Berezitsky

[Jeff]

Congrats, Ed!

On Thu, Jan 3, 2019 at 9:13 AM Ed B  wrote:

> Thank you all!!!
> :D
>
> Ed Berezitsky
>
> On Thu, Jan 3, 2019 at 8:02 AM Jeremy Dyer  wrote:
>
> > Congratulations Ed!
> >
> > On Thu, Jan 3, 2019 at 6:52 AM Mike Thomsen 
> > wrote:
> >
> > > Congratulations!
> > >
> > > On Thu, Jan 3, 2019 at 6:50 AM Otto Fowler 
> > > wrote:
> > >
> > > > Congratulations Ed!
> > > >
> > > >
> > > > On January 2, 2019 at 21:36:45, Tony Kurc (tk...@apache.org) wrote:
> > > >
> > > > On behalf of the Apache NiFI PMC, I am very pleased to announce that
> Ed
> > > has
> > > > accepted the PMC's invitation to become a committer on the Apache
> NiFi
> > > > project. We greatly appreciate all of Ed's hard work and generous
> > > > contributions to the project. We look forward to his continued
> > > involvement
> > > > in the project.
> > > >
> > > > Ed has been contributing code to the project through most of 2018, in
> > > areas
> > > > such as HBase, HDFS, and fixing some long standing bugs. Also, many
> of
> > > you
> > > > have had the pleasure of interacting with him on the dev and users
> > > mailing
> > > > lists, epitomizing The Apache Way.
> > > >
> > > > Welcome and congratulations!
> > > >
> > > > Tony
> > > >
> > >
> >
>
>
> --
> Edward Berezitsky
> bdes...@gmail.com
>

Re: [EXT] [discuss] release apache nifi 1.9.0

[Jeff]

https://issues.apache.org/jira/browse/NIFI-5575 (
https://github.com/apache/nifi/pull/3252) would be a good addition in
1.9.0, in my opinion.  It attempts to address PutHDFS ignoring the file
mask default value from hdfs-site.xml when no default mask is specified in
the PutHDFS configuration.

I can do the review on it, if we want to include it.

On Thu, Feb 7, 2019 at 1:17 AM Andy LoPresto  wrote:

> Ah, sorry, I misunderstood. You are not the PR author; you want those same
> comments to be addressed.
>
> Andy LoPresto
> alopre...@apache.org
> alopresto.apa...@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> > On Feb 7, 2019, at 5:16 PM, Andy LoPresto  wrote:
> >
> > Charlie, I see comments from Matt Burgess (especially considering
> exceptions & stack traces) from 9 days ago that don’t appear to be
> addressed. Were you waiting for additional information? Am I missing
> something? Thanks.
> >
> >
> > Andy LoPresto
> > alopre...@apache.org 
> > alopresto.apa...@gmail.com
> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >
> >> On Feb 7, 2019, at 3:24 PM, Charlie Meyer <
> charlie.me...@civitaslearning.com  charlie.me...@civitaslearning.com>> wrote:
> >>
> >> Hoping I am not too late to the party, but it would be fantastic if
> >> https://github.com/apache/nifi/pull/3253 <
> https://github.com/apache/nifi/pull/3253> could have the comments on it
> >> addressed and included the 1.9 release
> >>
> >> -Charlie
> >>
> >> On Wed, Feb 6, 2019 at 9:08 PM Joe Witt  joe.w...@gmail.com>> wrote:
> >>
> >>> MarkB - i dont have any issue with that being include but we need to
> find
> >>> someone familiar enough with that to review it.  If that happens in the
> >>> next couple days then great.  Any takers familiar with JMS and that
> class?
> >>> The changes are few but the implications are unclear to me.
> >>>
> >>> Thanks
> >>>
> >>> On Wed, Feb 6, 2019 at 9:55 PM Mark Bean  > wrote:
> >>>
>  I have an open PR I'd like included in 1.9. It's a very simple one.
> 
>  https://github.com/apache/nifi/pull/3053 <
> https://github.com/apache/nifi/pull/3053>
> 
>  Thanks,
>  Mark
> 
>  On Wed, Feb 6, 2019 at 7:43 PM Joe Witt  wrote:
> 
> > i have that one ready to go andy.  ill take care of the dev docs one.
> >
> > thanks
> >
> > On Wed, Feb 6, 2019, 7:24 PM Andy LoPresto  >>> wrote:
> >
> >> I also want to merge https://github.com/apache/nifi/pull/3283 <
> https://github.com/apache/nifi/pull/3283> <
> >> https://github.com/apache/nifi/pull/3283 <
> https://github.com/apache/nifi/pull/3283>> so we can get the
> >>> Developer
> >> Guide more visibility in the docs. Will do today.
> >>
> >> Andy LoPresto
> >> alopre...@apache.org 
> >> alopresto.apa...@gmail.com
> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >>
> >>> On Feb 7, 2019, at 10:12 AM, Ed B  wrote:
> >>>
> >>> Haha, just using this opportunity to pull attention and get
> >>> reviewer
>  :D
> >>>
> >>> But totally agreed, it's time for 1.9 to be finalized.
> >>>
> >>> Thanks,
> >>> Ed.
> >>>
> >>>
> >>> On Wed, Feb 6, 2019 at 5:08 PM Joe Witt  >
> >>> wrote:
> >>>
>  Thanks Andy sounds good.
> 
>  EdB: Sure we just need to find a reviewer.  There are tons of good
>  prs
> >> out
>  there its just about getting good review cycles/progress.
> 
>  I'd like to shoot for early next week RC beginning if we can pull
> >>> it
> >> off.
> 
>  Thanks
>  Joe
> 
>  On Wed, Feb 6, 2019 at 5:07 PM Andy LoPresto <
> >> alopresto.apa...@gmail.com >
>  wrote:
> 
> > The PR for adding HTTP headers is complete, I just need to merge
>  it.
> >> Will
> > do today.
> >
> > Andy LoPresto
> > alopre...@apache.org 
> > alopresto.apa...@gmail.com
> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D
> >>> EF69
> >
> >> On Feb 7, 2019, at 08:23, Ed B  bdes...@gmail.com>> wrote:
> >>
> >> if not too late, would be nice to have JNDI bug fix included in
>  new
> > release:
> >>
> >> "JMS Connection Fails After JMS servers Change behind JNDI"
> >> https://issues.apache.org/jira/browse/NIFI-5869 <
> https://issues.apache.org/jira/browse/NIFI-5869>
> >> PR is available:
> >> https://github.com/apache/nifi/pull/3261
> >>
> >> Thank you,
> >> Ed.
> >>
> >>
> >>> On Wed, Feb 6, 2019 at 1:13 PM Joe Witt 
> > wrote:
> >>>
> >>> thanks for bumping that pete

Re: [EXT] [discuss] release apache nifi 1.9.0

[Jeff]

Joe,

Yes, I'll review it today.

On Mon, Feb 11, 2019, 8:44 AM Joe Witt  wrote:

> Jeff - i added NIFI-5575 to 1.9.0 fix version. Are you going to
> review/merge that then?
>
> There are 5 remaining JIRAs tagged to 1.9.0 that aren't obviously puntable
> and nearing completion of review/merge.  Will monitor/poke those with
> intent to kick off RC once they're all done.
>
> Thanks
>
> On Fri, Feb 8, 2019 at 3:38 PM Jeff  wrote:
>
> > https://issues.apache.org/jira/browse/NIFI-5575 (
> > https://github.com/apache/nifi/pull/3252) would be a good addition in
> > 1.9.0, in my opinion.  It attempts to address PutHDFS ignoring the file
> > mask default value from hdfs-site.xml when no default mask is specified
> in
> > the PutHDFS configuration.
> >
> > I can do the review on it, if we want to include it.
> >
> > On Thu, Feb 7, 2019 at 1:17 AM Andy LoPresto 
> wrote:
> >
> > > Ah, sorry, I misunderstood. You are not the PR author; you want those
> > same
> > > comments to be addressed.
> > >
> > > Andy LoPresto
> > > alopre...@apache.org
> > > alopresto.apa...@gmail.com
> > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> > >
> > > > On Feb 7, 2019, at 5:16 PM, Andy LoPresto 
> > wrote:
> > > >
> > > > Charlie, I see comments from Matt Burgess (especially considering
> > > exceptions & stack traces) from 9 days ago that don’t appear to be
> > > addressed. Were you waiting for additional information? Am I missing
> > > something? Thanks.
> > > >
> > > >
> > > > Andy LoPresto
> > > > alopre...@apache.org <mailto:alopre...@apache.org>
> > > > alopresto.apa...@gmail.com
> > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> > > >
> > > >> On Feb 7, 2019, at 3:24 PM, Charlie Meyer <
> > > charlie.me...@civitaslearning.com  > > charlie.me...@civitaslearning.com>> wrote:
> > > >>
> > > >> Hoping I am not too late to the party, but it would be fantastic if
> > > >> https://github.com/apache/nifi/pull/3253 <
> > > https://github.com/apache/nifi/pull/3253> could have the comments on
> it
> > > >> addressed and included the 1.9 release
> > > >>
> > > >> -Charlie
> > > >>
> > > >> On Wed, Feb 6, 2019 at 9:08 PM Joe Witt   > > joe.w...@gmail.com>> wrote:
> > > >>
> > > >>> MarkB - i dont have any issue with that being include but we need
> to
> > > find
> > > >>> someone familiar enough with that to review it.  If that happens in
> > the
> > > >>> next couple days then great.  Any takers familiar with JMS and that
> > > class?
> > > >>> The changes are few but the implications are unclear to me.
> > > >>>
> > > >>> Thanks
> > > >>>
> > > >>> On Wed, Feb 6, 2019 at 9:55 PM Mark Bean  > > <mailto:mark.o.b...@gmail.com>> wrote:
> > > >>>
> > > >>>> I have an open PR I'd like included in 1.9. It's a very simple
> one.
> > > >>>>
> > > >>>> https://github.com/apache/nifi/pull/3053 <
> > > https://github.com/apache/nifi/pull/3053>
> > > >>>>
> > > >>>> Thanks,
> > > >>>> Mark
> > > >>>>
> > > >>>> On Wed, Feb 6, 2019 at 7:43 PM Joe Witt 
> wrote:
> > > >>>>
> > > >>>>> i have that one ready to go andy.  ill take care of the dev docs
> > one.
> > > >>>>>
> > > >>>>> thanks
> > > >>>>>
> > > >>>>> On Wed, Feb 6, 2019, 7:24 PM Andy LoPresto  > > >>> wrote:
> > > >>>>>
> > > >>>>>> I also want to merge https://github.com/apache/nifi/pull/3283 <
> > > https://github.com/apache/nifi/pull/3283> <
> > > >>>>>> https://github.com/apache/nifi/pull/3283 <
> > > https://github.com/apache/nifi/pull/3283>> so we can get the
> > > >>> Developer
> > > >>>>>> Guide more visibility in the docs. Will do today.
> > > >>>>>>
> > > >>>>>> Andy LoPresto
> > > >>>>>

Re: [VOTE] Release Apache NiFi 1.9.0 (rc2)

[Jeff]

+1 (binding) Release this package as nifi-1.9.0

Ran through the release guide, using Ubuntu 18.10 and jdk 1.8.0_201

Used a flow to test:
  - PutHDFS fs.permissions.umask-mode usage
  - Kerberized DBCPConnectionPool
  - Usage of ExecuteSql (with Kerberized DBCPConnectionPool) and PutHDFS to
insert data into Impala
  - Kerberized Hive 1.1 processors

Everything worked as expected.  Thanks a lot to everyone that contributed
to this release, and to Joe Witt for handing RM duties!

I ran into an issue with templates in which DistributedMapCacheServer is
not exported to a template.  After searching the Apache NiFi JIRAs, I found
an issue [1] that was reported at the end of 2015.  Since this is a
pre-existing issue, I'm still +1 for RC2.

[1] https://issues.apache.org/jira/browse/NIFI-1293

- Jeff

On Sat, Feb 16, 2019 at 10:50 PM Joe Witt  wrote:

> Hello,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> nifi-1.9.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1136
>
> The Git tag is nifi-1.9.0-RC2
> The Git commit ID is 45bb53d2aafd6ec5cb6bb794b3f7f8fc8300a04b
>
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=45bb53d2aafd6ec5cb6bb794b3f7f8fc8300a04b
>
> Checksums of nifi-1.9.0-source-release.zip:
> SHA256: f8d2987a98903f0c00c50677f3a6ad361e417c6021f5179280cbe9ca838695da
> SHA512:
>
> 2e77c420f932514417693584b4708a534df398e344dac7c1471f55cc382b7493d73b10ebc0d9e58562eb989c1f0b72980d6d18a2555883267f0bc08f092f30fe
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 160 issues were closed/resolved for this release:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12344357
>
> Release note highlights can be found here:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.9.0
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.9.0-rc2/
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test. Then please vote:
>
> [ ] +1 Release this package as nifi-1.9.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>

Re: Support for JDK 11

[Jeff]

Mike,

Due to other commitments, work on building NiFi on Java 11 was
de-prioritized, temporarily.  I'm picking up where I left off.  Here's a
short list of where we're at:
* https://issues.apache.org/jira/browse/NIFI-5820 for running NiFi on Java
11 that was built with Java 8.  I need to rebase the PR [1] on master,
though.
* My branch with the updated POMs and code compiles on Java 11, but some of
the tests are failing.  There are more code changes I need to make, among
them developing/integrating a replacement for dropwizard's JVM metrics
library, in which the version that can run on Java 9+ no longer has the
VirtualMachineMetrics class that some of our reporting tasks use.

I think it's close to being able to open a WIP PR so that live testing can
start, but I'd like some more time to at least get the remaining unit tests
passing.

[1] https://github.com/apache/nifi/pull/3174

On Sat, Mar 9, 2019 at 8:43 AM Mike Thomsen  wrote:

> I know there's a lot of work in progress here, but how far along are we to
> the point where users can build and deploy NiFi on JDK 11?
>
> I'm more than happy to jump in and help on reviews and making changes to
> move forward. Any chance we can push this for 1.10?
>
> Thanks,
>
> Mike
>

[DISCUSS] NiFi and Java 11

[Jeff]

I'm reaching out to the community today to propose a plan for moving
forward with NiFi on Java 11.

There are currently two PRs that deal with Java 11 compatibility.  The
first PR allows NiFi built on Java 8 to be run on Java 11 [1].  The second
PR allows NiFi to be built on Java 11 and run on Java 11 [2].  There are a
lot of changes in the second PR, and it will require a lot of testing due
to the breadth of changes from dependency upgrades.  Please take a look at
both of these PRs.  The earlier we can get feedback, the sooner we can get
Java 11 compatibility in an Apache NiFi release.

I would to discuss with the community the following plan for upcoming NiFi
releases as they pertain to Java 11:

For the 1.x release line, from either 1.10 or 1.11 (depending on when these
two PRs are merged to master) and onward, NiFi will be compatible with both
Java 8 and 11 for building and running NiFi:

   - Build on Java 8, run on Java 8
   - Build on Java 8, run on Java 11
   - Build on Java 11, run on Java 11

The 1.x release line will contain convenience builds for Java 8, and will
NOT contain convenience builds created from Java 11.  Users that want to
run NiFi 1.1x.y on Java 11 with Java 11 bytecode will have to build from
the released source code.

Once the Java 11 compatibility features [1] [2] are merged to master, PR
reviews will require building and testing with Java 8 and Java 11 before
the PR is merged to master. We will add a new build to our Travis CI
instance to cover building on Java 11, which should help with the PR
process, since build status from Appveyor and Travis are shown on the PR.
This will allow an easier transition to the NiFi 2.x release line.

For the 2.x release line, the minimum Java version would be Java 11, at
which point developers would be able to start using Java 11 language
features.  Java 12 is not designated to be an LTS release, with its support
ending in September 2019.  I would not recommend making our minimum Java
version a non-LTS Java release unless we are prepared to do NiFi releases
to update the minimum Java version before the support for that version of
Java ends.

Taking this approach means we can reasonably allow users to run on Java 11
in the 1.x release line, ensure that we do not add commits which are not
Java 11 compatible, and make it easier to transition into the 2.x line for
NiFi.

- Jeff

[1] https://github.com/apache/nifi/pull/3174
[2] https://github.com/apache/nifi/pull/3404

Re: [DISCUSS] NiFi and Java 11

[Jeff]

I created a JIRA [1] to track the developer documentation changes.  Andy,
thanks for pointing out the specific links; I've added them to the JIRA.

[1] https://issues.apache.org/jira/browse/NIFI-6187

On Wed, Apr 3, 2019 at 11:23 PM Sivaprasanna 
wrote:

> Yep. Adding to GitHub template is a great idea.
>
> On Thu, 4 Apr 2019 at 5:27 AM, Andy LoPresto  wrote:
>
> > We should add that to the Developer Guide [1], Contributor Guide [2],
> note
> > it in the Migration Guide [3] when it takes effect, and include it in the
> > GitHub PR template as well.
> >
> > [1] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html <
> > https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html>
> > [2] https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide <
> > https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide>
> > [3] https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance
> <
> > https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance>
> >
> >
> > Andy LoPresto
> > alopre...@apache.org
> > alopresto.apa...@gmail.com
> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >
> > > On Apr 3, 2019, at 1:16 PM, Sivaprasanna 
> > wrote:
> > >
> > > Sounds good to me.
> > >
> > > So contributors who bring in new changes are expected to write the code
> > > that should be compatible with both Java 8 as well as Java 11, right?
> > >
> > > Do we see anything that we should have in our site that would help the
> > > contributors/users with this change? Something like a document that
> > > explains best practices which a developer should follow so that the
> > change
> > > which the developer is bringing in should run fine in both 8 and 11.
> > >
> > > -
> > > Sivaprasanna
> > >
> > > On Thu, 4 Apr 2019 at 12:18 AM, Pierre Villard <
> > pierre.villard...@gmail.com>
> > > wrote:
> > >
> > >> Sounds good to me as well. Given the latest news on Java 8, having
> the 2
> > >> PRs merged in would be great!
> > >>
> > >> Thanks,
> > >> Pierre
> > >>
> > >> Le mer. 3 avr. 2019 à 20:34, Joe Witt  a écrit :
> > >>
> > >>> Jeff
> > >>>
> > >>> This seems very reasonable and thorough to me.
> > >>>
> > >>> The only short term implication that we'd have to buy into then is
> that
> > >> PRs
> > >>> going forward need to be able to build on both Java 8 and 11 which
> > seems
> > >> a
> > >>> very fair way to bridge toward moving to Java 11 as the base
> > requirement
> > >> in
> > >>> the next major release of NiFi (2.x).
> > >>>
> > >>> Thanks
> > >>> Joe
> > >>>
> > >>> On Wed, Apr 3, 2019 at 2:19 PM Jeff  wrote:
> > >>>
> > >>>> I'm reaching out to the community today to propose a plan for moving
> > >>>> forward with NiFi on Java 11.
> > >>>>
> > >>>> There are currently two PRs that deal with Java 11 compatibility.
> The
> > >>>> first PR allows NiFi built on Java 8 to be run on Java 11 [1].  The
> > >>> second
> > >>>> PR allows NiFi to be built on Java 11 and run on Java 11 [2].  There
> > >> are
> > >>> a
> > >>>> lot of changes in the second PR, and it will require a lot of
> testing
> > >> due
> > >>>> to the breadth of changes from dependency upgrades.  Please take a
> > look
> > >>> at
> > >>>> both of these PRs.  The earlier we can get feedback, the sooner we
> can
> > >>> get
> > >>>> Java 11 compatibility in an Apache NiFi release.
> > >>>>
> > >>>> I would to discuss with the community the following plan for
> upcoming
> > >>> NiFi
> > >>>> releases as they pertain to Java 11:
> > >>>>
> > >>>> For the 1.x release line, from either 1.10 or 1.11 (depending on
> when
> > >>> these
> > >>>> two PRs are merged to master) and onward, NiFi will be compatible
> with
> > >>> both
> > >>>> Java 8 and 11 for building and running NiFi:
> > >>>>
> > >>>>   - Build on Java 8, ru

Re: [DISCUSS] NiFi and Java 11

[Jeff]

Good idea Andy.

I use the community edition of Intellij IDEA (currently 2019.1), and I have
the project's SDK set to Java 11.  I have Intellij IDEA configured to use
an external installation of Maven (3.5.4, installed via homebrew) and when
building with Maven in Intellij IDEA, Maven uses the project-configured
SDK, ultimately building with Java 11.

I use jEnv to switch between various JDK versions when building or running
from the command line.  When using jEnv, after the desired Java
installations are added to jEnv, the "maven" plugin for jEnv should be
activated, so that Maven uses the Java installation set via the jEnv
utility.  Without the "maven" plugin activated, Maven will use the Java
installation configured with the system.  This can be done by running:

*$ jenv enable-plugin maven*
maven plugin activated

You can verify Maven is using the correct Java installation by doing the
following:

*$ jenv versions   # list configured installations*
* system (set by /Users/jeff/.jenv/version)
  1.8
  1.8.0.192
  11.0
  11.0.2
  openjdk64-11.0.2
  oracle64-1.8.0.192
*$ jenv shell 11.0# set this current shell to use "11.0" which is alias
(along with "11.0.2") of the openjdk64-11.0.2 installation*
*$ mvn -v   # list maven and java version info*
Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe;
2018-06-17T14:33:14-04:00)
Maven home: /usr/local/Cellar/maven/3.5.4/libexec
Java version: 11.0.2, vendor: Oracle Corporation, runtime:
/Library/Java/JavaVirtualMachines/openjdk-11.0.2.jdk/Contents/Home
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.14.3", arch: "x86_64", family: "mac"

There's a profile set up for building on Java 11, called "jigsaw", which
automatically activates when a JDK version greather than 1.8 is used to
build NiFi.

I build NiFi with Maven in Intellij IDEA and on the command line.  One of
the goals I had for Java 11 build compatibility was to keep it simple.
Make Java 11 available to Intellij IDEA, and/or maven, and build NiFi as
usual, letting the auto-activating profile handle configuring build
plugins, extra dependencies (such as JAXB, annotation/activation, etc).

On a side note, while trying to build the WIP Java 11 PR with Java 8, I did
notice some test failures.  I will address these ASAP.

- Jeff

On Thu, Apr 4, 2019 at 6:12 PM Andy LoPresto  wrote:

> Jeff,
>
> I think it would also be valuable to the community if you could document
> your tooling/setup for verifying the Java 11 build and performance, as this
> may be new to some community members. This doesn’t have to be an
> endorsement of specific products, but I know things like jEnv [1] can help
> with running multiple JVM/JREs at the same time. I am curious to see how
> you configured your IDE of choice and command-line to work with both
> versions. Thanks.
>
>
> [1] http://www.jenv.be/ <http://www.jenv.be/>
>
> Andy LoPresto
> alopre...@apache.org
> alopresto.apa...@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> > On Apr 4, 2019, at 10:12 AM, Kevin Doran  wrote:
> >
> > Jeff,
> >
> > Thanks for all your work on this and for explaining the path forward.
> > I also support this proposal.
> >
> > I suggest we take the same basic approach for NiFi Registry. That is:
> >
> > - For the current NiFi Registry 0.x line, we maintain Java 8 and 11
> > build compatibility
> > - We perform Java 8 builds (that can run on 8 and 11) for our
> > convenience binaries
> > - We plan to bump the major version at some point in the future in
> > step with NiFi which will move us to requiring Java 11 for Registry as
> > well
> >
> > I've looked at your PRs, and I think getting NiFi Registry to the
> > point that it can run on Java 11 will be much easier than NiFi was
> > given it is a much smaller and simpler project, and it will mostly
> > require a subset of the changes you made for NiFi (Groovy, JaxB
> > dependencies for example).
> >
> > Thanks,
> > Kevin
> >
> > On Thu, Apr 4, 2019 at 12:54 PM Matt Gilman 
> wrote:
> >>
> >> Jeff,
> >>
> >> I think this is a great approach that should help us transition to and
> in
> >> the long run, adopt Java 11. The staggered approach should allow us to
> >> address concerns and any issues we may encounter as we go over time.
> >>
> >> Thanks for putting these together!
> >>
> >> Matt
> >>
> >> On Thu, Apr 4, 2019 at 2:19 AM Jeff  wrote:
> >>
> >>> I created a JIRA [1] to track the developer documentation changes.
> Andy,
> >>> thanks for pointing o

Re: [VOTE] Release Apache NiFi 1.9.2 (rc2)

[Jeff]

+1, binding

- Performed release guide steps, verified checksums, signature, git commit
hash
- Full build with contrib-check and include-grpc profiles completed
successfully with tests
- Started a standalone instance with some tests flows

Thanks for RMing, Joe!


On Thu, Apr 4, 2019 at 12:11 PM Joe Witt  wrote:

> Hello,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> nifi-1.9.2.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-
>
> The Git tag is nifi-1.9.2-RC2
> The Git commit ID is 11320acfbaab1a3579bd8f7545473d6984472d77
>
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=11320acfbaab1a3579bd8f7545473d6984472d77
>
> Checksums of nifi-x.y.z-source-release.zip:
> SHA256: 1f666c30171e2b7be847a3a852e9176420fd0176537f3ce6428b786a6d24f456
> SHA512:
>
> f65b81976a9811017348f9e8340faff3a29b029878845dd509db98eb7449752c8ef0bdbf5d1f67635636bb08be28086015442f546821eebf76b499cebf1a386b
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 11 issues were closed/resolved for this release:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12345213
>
> Release note highlights can be found here:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.9.2
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test. Then please vote:
>
> [ ] +1 Release this package as nifi-1.9.2
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

[Jeff]

In the same category of challenges that Peter pointed out, it might be
difficult for Travis to build the "framework" and "extensions" projects if
there are changes in a PR that affect both projects.

Is there a good way in Travis to have the workspace/maven repo shared
between projects in a single build?

It's probably always in the direction of the extensions project needing
something new to be added to the framework project rather than the other
way around, but it'll be tricky to get that working right in Travis if it's
not possible to set up the Travis build to know it needs to deploy the
framework project artifacts into a maven repo that the extension project
will use.

One way might be to make sure that changes to the framework project must be
in master before the extensions project can make use of them, but that
would require a "default master" build for the framework project which
builds master after each commit, and deploys the build artifacts to a
persistent maven repo that the extension project builds can access.  It
also makes project-spanning change-sets take longer to review and get fully
committed to master.

On Thu, May 30, 2019 at 11:23 AM Peter Wicks (pwicks) 
wrote:

> One more "not awesome" would be that core changes that affect extensions
> will be a little harder to test. If I make a core change that changes the
> signature of an interface/etc... I'll need to do some extra work to make
> sure I don't break extensions that use it.
>
> Still worth it, just one more thing to mention.
>
> -Original Message-
> From: Joe Witt 
> Sent: Thursday, May 30, 2019 9:19 AM
> To: dev@nifi.apache.org
> Subject: [EXT] [discuss] Splitting NiFi framework and extension repos and
> releases
>
> Team,
>
> We've discussed this a bit over the years in various forms but it again
> seems time to progress this topic and enough has changed I think to warrant
> it.
>
> Tensions:
> 1) Our build times take too long.  In travis-ci for instance it takes 40
> minutes when it works.
> 2) The number of builds we do has increased.  We do us/jp/fr builds on
> open and oracle JDKs.  That is 6 builds.
> 3) We want to add Java 11 support such that one could build with 8 or 11
> and the above still apply.  The becomes 6 builds.
> 4) With the progress in NiFi registry we can now load artifacts there and
> could pull them into NiFi.  And this integration will only get better.
> 5) The NiFi build is too huge and cannot grow any longer or else we cannot
> upload convenience binaries.
>
> We cannot solve all the things just yet but we can make progress.  I
> suggest we split apart the NiFi 'framework/application' in its own release
> cycle and code repository from the 'nifi extensions' into its own
> repository and release cycle.  The NiFi release would still pull in a
> specific set of extension bundles so to our end users at this time there is
> no change. In the future we could also just stop including the extensions
> in nifi the application and they could be sourced at runtime as needed from
> the registry (call that a NiFi 2.x thing).
>
> Why does this help?
> - Builds would only take as long as just extensions take or just core/app
> takes.  This reduces time for each change cycle and reduces load on
> travis-ci which runs the same tests over and over and over for each pull
> request/push regardless of whether it was an extension or core.
>
> - It moves us toward the direction we're heading anyway whereby extensions
> can have their own lifecycle from the framework/app itself.
>
> How is this not awesome:
> - Doesn't yet solve for the large builds problem.  I think we'll get there
> with a NiFi 2.x release which fully leverages nifi-registry for retrieval
> of all extensions.
> - Adds another 'thing we need to do a release cycle for'.  This is
> generally unpleasant but it is paid for once a release cycle and it does
> allow us to release independently for new cool extensions/fixes apart from
> the framework itself.
>
> Would be great to hear others thoughts if they too feel it is time to make
> this happen.
>
> Thanks
> Joe
>

Re: [ANNOUNCE] New Apache NiFi PMC member Peter Wicks

[Jeff]

Welcome to the PMC, Peter!  Congrats!

On Thu, May 30, 2019 at 2:45 PM Tony Kurc  wrote:

> Congratulations Peter!!
>
> On Thu, May 30, 2019 at 11:21 AM Aldrin Piri  wrote:
>
> > NiFi Community,
> >
> > On behalf of the Apache NiFi PMC, I am pleased to announce that Peter
> Wicks
> > has accepted the PMC's invitation to join the Apache NiFi PMC.
> >
> > Peter's contributions have been plentiful in code, community, reviews and
> > discussion after becoming a committer in November 2017.  His impact
> across
> > NiFi has lead to improvements surrounding Kerberos, GetFile, ListFile,
> > Clustering, Node Offload, Recordset Writers, HDFS, and Database related
> > processors among others.
> >
> > Thank you for all your contributions and welcome to the PMC, Peter!
> >
> > --aldrin
> >
>

NiFi and Mockito 2.28.2

[Jeff]

NiFi Developers,

Recently, NIFI-6360 [1] was resolved with PR 3533 [2] merged to master.
All NiFi components were upgraded to use Mockito version 2.28.2, including
updating those components to use mockito-core instead of mockito-all since
the mockito-all distribution was discontinued in Mockito 2.

The Mockito dependency upgrade was one of several PRs (with a few more on
the way) needed to support building NiFi on Java 11.

For those of you with PRs open that reference Mockito, please rebase onto
the current master.  If you have specific issues with the Mockito upgrade,
I'd be happy to help you resolve them.

[1] https://issues.apache.org/jira/browse/NIFI-6360
[2] https://github.com/apache/nifi/pull/3533

Upgrading Groovy to 2.5.4

[Jeff]

NiFi Devs,

I've created a draft PR [1], mostly from code in the draft Java 11 PR [2]
to drive the upgrade of NiFi's Groovy dependencies to version 2.5.4.  If
you're interested in NiFi+Groovy, please take a look at it.  Several POMs
have been updated to clean up some configurations of maven-compiler-plugin,
a migration from groovy-all to individual modules, etc.  So far, the build
and tests are passing.  More live testing is required, but on a first pass,
it's looking promising.

There's a regression of an issue fixed in Groovy 2.5.4 that has resurfaced
in Groovy 2.5.7, so for now the PR uses Groovy 2.5.4.

Mike Thomsen, we can coordinate to bring in necessary changes from your PR
[3] and general review, and commence testing.

[1] https://github.com/apache/nifi/pull/3547
[2] https://github.com/apache/nifi/pull/3404
[3] https://github.com/apache/nifi/pull/3015

Re: [board report] Apache NiFi - July 2019

[Jeff]

Russell,

NIFI-5176 [1] is tracking Java 11 build compatibility for NiFi.  PR-3404
[2] is a work-in-progress PR that I welcome review and testing from anyone
in the community.  From PR 3404, several other PRs have been extracted to
prepare NiFi for Java 11 compatibility, and the JIRAs driving those PRs are
linked to NIFI-5176.

The current release of NiFi (and going back to NiFi 1.7.0) although built
with Java 8, will run on Java 11.

if you have any questions or concerns with NiFi on built or run on Java 11,
I'd be happy to discuss in a separate thread on the dev list.

[1] https://issues.apache.org/jira/browse/NIFI-5176
[2] https://github.com/apache/nifi/pull/3404

On Wed, Jul 10, 2019 at 5:28 PM Russell Bateman 
wrote:

> I mulled over whether it's appropriate to ask within this context. I
> guess I'll ask. Slap me if I have badly chosen, but where is NiFi
> respective to "modern" JDK versions? Is that too much detail for the
> audience of this periodic report?
>
> Thanks.
>
> On 7/10/19 11:15 AM, Joe Witt wrote:
> > Team,
> >
> > I was running late so submitted the report already.  Here is what I sent
> to
> > the board for Apache NiFi July 2019 report.  Great work and great
> progress
> > all!
> >
> >
> >
> > ## Description:
> >   - Apache NiFi is an easy to use, powerful, and reliable system to
> process
> > and
> > distribute data.
> >   - Apache NiFi MiNiFi is an edge data collection agent built to
> seamlessly
> > integrate with and leverage the command and control of NiFi. There
> are
> > both
> > Java and C++ implementations.
> >   - Apache NiFi Registry is a centralized registry for key configuration
> > items
> > including flow versions, assets, and extensions for Apache NiFi and
> > Apache
> > MiNiFi.
> >   - Apache NiFi Nar Maven Plugin is a release artifact used for
> supporting
> > the
> > NiFi classloader isolation model.
> >   - Apache NiFi Flow Design System is a theme-able set of high quality UI
> > components and utilities for use across the various Apache NiFi web
> > applications in order to provide a more consistent user experience.
> >
> > ## Issues:
> >   - There are no issues requiring board attention at this time.
> >
> > ## Activity:
> >   - Released Apache NiFi Registry 0.4.0 which allows storage of
> extensions
> > which
> > is a critical step towards us breaking apart the monolithic release
> of
> > today
> > resulting in smaller binaries in Apache infra a mirrors and will also
> > bring
> > a better user experience for updating live running NiFi clusters and
> > operations in container based environments.
> >   - The community is working the release preparation and voting process
> for
> > Apache NiFi MiNiFi CPP 0.6.1.
> >   - Apache NiFi 0.10.0 is progressing nicely with nearly 200 JIRAs
> already
> > included.  This brings powerful features such as sourcing extensions
> from
> > the latest NiFi Registry at runtime, far better model for
> paramaterized
> > version controlled flows, Java 11 compatibility, and much more.
> >
> > ## Health report:
> >   - Health of the community remains strong with many active release
> lines,
> > feature development, active user and developer base including new
> > participants and continued participants.
> >   - We see considerable commentary on Apache NiFi in the form of meetups,
> > conferences, training, and talks including from folks at Google,
> Cloudera
> > and others. A recent tweet from a Ford Motor Company employee says it
> > best
> > "Love, love, love @apachenifi   Using MiNiFi opens up IOT innovation
> in
> > amazing ways.  Every IOT device manufacturer, including automotive,
> > should consider opening up API's to let developers do their thing!
> > Bringing value to your product! "
> >
> > ## PMC changes:
> >
> >   - Currently 30 PMC members.
> >   - Peter Wicks was added to the PMC on Wed May 29 2019
> >
> > ## Committer base changes:
> >
> >   - Currently 43 committers.
> >   - Arpad Boda was added as a committer on Thu May 23 2019
> >
> > ## Releases:
> >
> >   - Apache NiFi Registry 0.4.0 was released on Mon May 20 2019
> >
> > ## Mailing list activity:
> >
> >   - Activity on the mailing lists remains high with a mixture of new
> users,
> > contributors, and deeper more experienced users and contributors
> sparking
> > discussion and questions and filing bugs or new features.
> >
> >   - We do see a significant drop in users list usage while dev and issues
> > remains busy and even growing.  Meanwhile we see strong growth in our
> > slack channel and it is very user centric.
> >
> >   - Slack Channel Usage: apachenifi.slack.com
> >- 394 users currently in the room.  This has grown weekly.
> >
> >   - us...@nifi.apache.org:
> >  - 693 subscribers (up 18 in the last 3 months):
> >  - 447 emails sent to list (800 in previous quarter)
> >
> >   - dev@nifi.apache.org:
> >  - 443 subscribers (down -1 i

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

[Jeff]

oposed
> > > > > are
> > > > > > > > done under a single mono-repository model.
> > > > > > > >
> > > > > > > > If we split into multiple repositories, you have
> substantially
> > > > > > increased
> > > > > > > > the infra surface area. User account management overhead goes
> > up.
> > > > > > Support
> > > > > > > > from the infra team goes up. JIRA issue management goes up,
> > > > > > > > misfiled/miscategorized issues become common. It becomes
> > harder for
> > > > > > > > community members to interact and engage with the project,
> > steeper
> > > > > > > learning
> > > > > > > > curve for new contributors. There are more "side channel"
> > > > > conversations
> > > > > > > and
> > > > > > > > less transparency into the project as a whole. Git history is
> > much
> > > > > > harder
> > > > > > > > (or impossible) to follow across the entire project. Tracking
> > down
> > > > > bugs
> > > > > > > and
> > > > > > > > performing git blame or git bisect becomes hard.
> > > > > > > >
> > > > > > > > There's nothing really stopping all of these changes from
> > occurring
> > > > > in
> > > > > > > the
> > > > > > > > existing repo, we don't have to have a maven pom.xml in the
> > root of
> > > > > the
> > > > > > > > project repository. It's much easier for contributors to just
> > clone a
> > > > > > > > single repository, read the README at the root, and get
> > oriented to
> > > > > the
> > > > > > > > project layout.  Output artifacts can still be versioned
> > differently
> > > > > > (api
> > > > > > > > can have a different version from extensions).  "Splitting
> out"
> > > > > modules
> > > > > > > can
> > > > > > > > still happen in the mono-repository.  Jenkins and friends can
> > be
> > > > > taught
> > > > > > > the
> > > > > > > > project layout.
> > > > > > > >
> > > > > > > > tl;dr - The changes being proposed can be done in a single
> > > > > repository.
> > > > > > > > Splitting into multiple repositories is adding overhead on
> > multiple
> > > > > > > levels,
> > > > > > > > which might be a sneaky form of muda. [1]
> > > > > > > >
> > > > > > > > Thanks for reading,
> > > > > > > > Adam
> > > > > > > >
> > > > > > > > [1] https://dzone.com/articles/seven-wastes-software
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Jul 11, 2019 at 11:01 AM Otto Fowler <
> > > > > ottobackwa...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I agree that this looks great. I think Mike’s idea is worth
> > > > > > considering
> > > > > > > > as
> > > > > > > > > well. I would hope, that as part of this effort some
> thought
> > will
> > > > > be
> > > > > > > > given
> > > > > > > > > to enhancing the developer documentation around the modules
> > would
> > > > > be
> > > > > > > > given
> > > > > > > > > as well.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On July 10, 2019 at 18:15:21, Mike Thomsen (
> > mikerthom...@gmail.com
> > > > > )
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > I agree. It's very well thought out. One change to consider
> > is
> > > > > > > splitting
> > > > > > > > > the extensions further into two s

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

[Jeff]

.  As a
> > >>>>> many time RM I definitely dislike the mono repo construct as I
> > >> understand
> > >>>>> it to function.  I prefer repos per source release artifact where
> all
> > >>>>> source in that artifact is a function of the release. I am ok with
> > >>>>> different convenience binaries resulting from a single source
> release
> > >>>>> artifact though.
> > >>>>>
> > >>>>> Thanks
> > >>>>>
> > >>>>> On Fri, Jul 12, 2019 at 12:26 PM Adam Taft 
> > >> wrote:
> > >>>>>
> > >>>>>> I think the concerns around user management are valid, are they
> > >> not?
> > >>>>>> Overhead in JIRA goes up (assigning rights to users in JIRA is
> > >>>>>> multiplied).  Risk to new contributors is high, because each
> > >> isolated
> > >>>>>> repository has its own life and code contribution styles.  Maybe
> > >> the
> > >>>>> actual
> > >>>>>> apache infra involvement is low, but the negative effects of
> > >> community
> > >>>>> and
> > >>>>>> source code bifurcation goes up.
> > >>>>>>
> > >>>>>> Tagging in mono-repos is done by prefixing the name of the
> > >> component in
> > >>>>> the
> > >>>>>> tag name.  Your release sources are still generated from the
> > >> component
> > >>>>>> folder (not from the root).
> > >>>>>>
> > >>>>>> Modularization (as being proposed) is a good thing, but can be
> > >> done in a
> > >>>>>> single repository. It's not a requirement to split up the git
> > >> project to
> > >>>>>> get the benefits of modularization.  That's the point I'm hoping
> > >> is seen
> > >>>>> in
> > >>>>>> this.
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> On Fri, Jul 12, 2019 at 10:08 AM Joe Witt 
> > >> wrote:
> > >>>>>>
> > >>>>>>> to clarify user management for infra is not a prob.  it is an
> > >> ldap
> > >>>>> group.
> > >>>>>>>
> > >>>>>>> repo creation is self service as well amd group access is tied
> > >> to that.
> > >>>>>>>
> > >>>>>>> release artifact is the source we produce.  this is typically
> > >>>>> correlated
> > >>>>>> to
> > >>>>>>> a tag of the repo.  if we have all source in one repo it isnt
> > >> clear to
> > >>>>> me
> > >>>>>>> how we can maintain that.
> > >>>>>>>
> > >>>>>>> in any event im not making a statement of whether to do many
> > >> repos or
> > >>>>>> not.
> > >>>>>>> just correcting some potentially misleading claims.
> > >>>>>>>
> > >>>>>>> thanks
> > >>>>>>>
> > >>>>>>> On Fri, Jul 12, 2019, 12:01 PM Adam Taft 
> > >> wrote:
> > >>>>>>>
> > >>>>>>>> Just as a point of discussion, I'm not entirely sure that
> > >> splitting
> > >>>>>> into
> > >>>>>>>> multiple physical git repositories is actually adding any
> > >> value.  I
> > >>>>>> think
> > >>>>>>>> it's worth consideration that all the (good) changes being
> > >> proposed
> > >>>>> are
> > >>>>>>>> done under a single mono-repository model.
> > >>>>>>>>
> > >>>>>>>> If we split into multiple repositories, you have substantially
> > >>>>>> increased
> > >>>>>>>> the infra surface area. User account management overhead goes
> > >> up.
> > >>>>>> Support
> > >>>>>>>> from the infra team goes up. JIRA issue management goes up,
>

Re: Java 12 Compatibility

[Jeff]

Hamza,

This particular reflective access issue, as Edward mentioned, isn't
something you need to worry about.  The code that's causing that issue is
from accessing the Java 9+ Process API from code compiled on Java 8, and
was a necessary addition for getting NiFi built on Java 8 to run on Java 9+.

NIFI-5176 [1] is the task driving build compatibility for NiFi on Java 11
and it is very close to being out of draft status.  When building and
running NiFi on Java 11+, you won't see this particular warning.

[1] https://issues.apache.org/jira/browse/NIFI-5176

On Wed, Jul 17, 2019 at 11:08 AM Edward Armes 
wrote:

> So the warning here isn't something you need to worry about and you'll find
> it's quite a common one, on a lot of Java applications and its down to
> changes made in Java 9,
>
> The short reason for this, is that in Java 9 a new type of package was
> created a Module. A module is essentially a package of packages. The
> important thing to note here is that by default unless it is done
> explicitly then a module won't expose it's entire API by default even if a
> resource is marked a public, the important part here is that this
> restriction also applies to the Java reflection system as well.
>
> In Java 9 the standard Java API seems to have been implemented into modules
> and certain things are not exposed in the module definition. Now to get
> people onto Java 9+ one of things that was decided was, that by default the
> reflect system would allowed access unexposed API in modules  for now.
> However it is clear from various bits of documentation that in the future
> that this will change, and that specific JVM flags will need to be used to
> override the module export definition to expose packages. The intention of
> this flag seems to be that it is made clear that you are invoking an
> internal module that a developer of the module wasn't intending people to
> use, so when it breaks it's not developers fault.
>
> I would note what  I've said above is a very rough explanation and there is
> a lot more complexity and subtlety to how this works. At some point in the
> future I'm going to do some proper research into this, as I can see it in
> the future giving me and others the runaround.
>
> Like I said it's nothing to worry about and it's quite common as you will
> see this warning on anything that's using a version of Spring that pre-java
> 9 and using parts of Spring that use reflection on a Java 9 or newer run
> time.
>
> Edward
>
> On Wed, Jul 17, 2019 at 12:39 AM Mike Thomsen 
> wrote:
>
> > I believe those warnings are given with Java 11 as well. Java 12 is not
> > officially supported, but that's not to say it's incompatible with NiFi
> > since the delta between it and Java 11 is not that big. I would recommend
> > avoiding any experimental features bundled with Java 12 and stick to ones
> > that the OpenJDK team says are stable in Java 12. For production
> scenarios,
> > Java 8 or Java 11 would be strongly recommended over Java 12.
> >
> > On Tue, Jul 16, 2019 at 7:27 PM Hamza Mesbahi <
> hamzamesbahi1...@gmail.com>
> > wrote:
> >
> > > Hello,
> > >
> > > I've downloaded NiFi 1.9.2. and installed it on my Windows 10 using cmd
> > > window. I currently have Java 12 installed on my computer and
> executable
> > > from Path.
> > > When I input the following command: "Start run-nifi.bat" I get the
> usual
> > > gibberish but towards the end 10-15 lines this is what I get: WARNING:
> An
> > > illegal reflective access operation has occurred
> > > WARNING: Illegal reflective access by
> > > org.apache.nifi.bootstrap.util.OSUtils
> > > (file:/C:/Nifi/nifi-1.9.2/lib/bootstrap/nifi-bootstrap-1.9.2.jar) to
> > method
> > > java.lang.ProcessImpl.pid()
> > > WARNING: Please consider reporting this to the maintainers of
> > > org.apache.nifi.bootstrap.util.OSUtils
> > > WARNING: Use --illegal-access=warn to enable warnings of further
> illegal
> > > reflective access operations
> > > WARNING: All illegal access operations will be denied in a future
> release
> > > 2019-07-16 15:50:26,289 WARN [main] org.apache.nifi.bootstrap.Command
> > > Failed to set permissions so that only the owner can read pid file
> > > C:\Nifi\NIFI-1~1.2\bin\..\run\nifi.pid; this may allows others to have
> > > access to the key needed to communicate with NiFi. Permissions should
> be
> > > changed so that only the owner can read this file
> > > 2019-07-16 15:50:26,293 WARN [main] org.apache.nifi.bootstrap.Command
> > > Failed to set permissions so that only the owner can read status file
> > > C:\Nifi\NIFI-1~1.2\bin\..\run\nifi.status; this may allows others to
> have
> > > access to the key needed to communicate with NiFi. Permissions should
> be
> > > changed so that only the owner can read this file
> > > 2019-07-16 15:50:26,321 INFO [main] org.apache.nifi.bootstrap.Command
> > > Launched Apache NiFi with Process ID 2112
> > > I've been doing some research and have found that for a while Nifi
> wasn't
> > > compatible wit

Re: CM of nar via toolkit cli and regisry

[Jeff]

Aaron,

New versions of NiFi are generally released after a member of the community
proposes a release and someone volunteers to perform the duties of the
release manager.  There is no predefined schedule or projected release
dates.  I think there are a few major feature JIRAs on which work is
currently being done, and once those are resolved/finished, at that point
someone may make proposal to create a release candidate for 1.10.0.

Regarding the error to which you are referring, does the
TestStandardMyService class make use of nifi-mock classes?  Is
TestStandardMyService within the "src/main" folder of your module?  If so,
you may want to move that class into "src/test", as I don't believe the
nifi-maven-nar plugin bundles test code when it creates the NAR.  From the
stacktrace, I see that TestStandardMyService makes a call to a
org.apache.nifi.util.MockControllerServiceIntitializationContext method,
which comes from nifi-mock.  If TestStandardMyService is included with the
rest of your "production" code, you might want to refactor your module so
that it is part of the "test" code.  That should remove the dependency on
nifi-mock.

On Mon, Jul 22, 2019 at 12:24 AM Aaron Rich  wrote:

> I updated  pom  for  a bundle with just a processor and was able to get nar
> generated and loaded into registry via cli.
>
> When I tried a bundle with a custom service though, I get failures in the
> test:
> java.lang.AbstractMethodError:
>
> org.apache.nifi.util.MockControllerServiceInitializationContext.getNodeTypeProvider()Lorg/apache/nifi/controller/NodeTypeProvider;
> at
>
> com.ngc.swordfish.nifi.multipart.TestStandardMyService.testService(TestStandardMyService.java:36)
>
> Do I need to updated nifi-mock to 1.10 also?
>
> Thanks.
>
> -Aaron
>
> On Sun, Jul 21, 2019 at 9:59 PM Aaron Rich  wrote:
>
> > Also,
> >
> > When is 1.10.0 planned for release?
> >
> > Thanks.
> >
> > -Aaron
> >
> >
> > On Sun, Jul 21, 2019 at 9:29 PM Aaron Rich  wrote:
> >
> >> Great.
> >>
> >> Thank you.
> >>
> >> Is the best way to force my nar to use the 1.10 nifi-api via adding it
> as
> >> dependency is main pom file?
> >>
> >> Assuming I still need the build plugin also?
> >>
> >> Thanks again.
> >>
> >> -Aaron
> >>
> >> On Sun, Jul 21, 2019 at 9:30 AM Bryan Bende  wrote:
> >>
> >>> Hello,
> >>>
> >>> The issue is that in order to correctly generate the extension manifest
> >>> with the new NAR plugin, it requires changes from nifi-api that are not
> >>> released yet.
> >>>
> >>> You should be able to build NiFi on the master branch, really just the
> >>> nifi-api module, doing a mvn clean install.
> >>>
> >>> Then in your NAR you’ll need to force it to use 1.10.0-SNAPSHOT of
> >>> nifi-api. Currently you are getting 1.9.2 because of the parent of
> >>> nifi-nar-bundles.
> >>>
> >>> Once we get 1.10.0 released then this won’t be an issue anymore.
> >>>
> >>> Thanks,
> >>>
> >>> Bryan
> >>>
> >>> On Sat, Jul 20, 2019 at 11:36 PM Aaron Rich 
> >>> wrote:
> >>>
> >>> > I forgot to include that when generating the .nar, I do get the
> >>> warnings
> >>> > of:
> >>> > [WARNING] Could not generate extensions' documentation
> >>> > org.apache.maven.plugin.MojoExecutionException: Failed to create
> >>> Extension
> >>> > Documentation
> >>> > ...
> >>> > Caused by: java.lang.NoSuchMethodException:
> >>> >
> >>> >
> >>>
> org.apache.nifi.documentation.xml.XmlDocumentationWriter.initialize(org.apache.nifi.components.ConfigurableComponent)
> >>> >
> >>> > I believe this is tied to not getting the right nfi-api version? But
> >>> the
> >>> > parent is set to 1.9.2 so not sure how that is happening.
> >>> >
> >>> > Thanks again.
> >>> >
> >>> > -Aaron
> >>> >
> >>> > On Sat, Jul 20, 2019 at 9:09 PM Aaron Rich 
> >>> wrote:
> >>> >
> >>> > > Hi,
> >>> > >
> >>> > > I'm trying to determine the best way to CM custom nar files for
> >>> sharing
> >>> > > between team. We are using nifi-registry for the flows and I saw
> >>> there
> >>> > was
> >>> > > a new capability via the toolkit cli for "upload-bundle".
> >>> > >
> >>> > > I'm trying to use that but have ran into a few issues:
> >>> > > 1) I first wasn't getting the META-INF/docs/ in the .nar. I had
> >>> built the
> >>> > > initial project from mvn archetype:generate with version 1.9.2. I
> >>> added:
> >>> > > 
> >>> > > 
> >>> > > 
> >>> > > org.apache.nifi
> >>> > > nifi-nar-maven-plugin
> >>> > > 1.3.1
> >>> > > true
> >>> > > 
> >>> > > 
> >>> > > 
> >>> > >
> >>> > > To the base pom.xml. It has the parent of:
> >>> > > 
> >>> > > org.apache.nifi
> >>> > > nifi-nar-bundles
> >>> > > 1.9.2
> >>> > > 
> >>> > >
> >>> > > That got the docs in the jar:
> >>> > >  0 Sat Jul 20 20:53:44 MDT 2019 META-INF/docs/
> >>> > > 72 Sat Jul 20 20:53:44 MDT 2019
> >>> META-INF/docs/extension-manifest.xml
> >>> > >  0 Sat Jul 20 20:53:44 MDT 2019
> META-INF/docs/additional-details/
> >>> > >
> >>> > > 2)I tried to use cli then with command:
> >>> > > ./bin/cli.sh registry upload-bundle --base

Re: CM of nar via toolkit cli and regisry

[Jeff]

Aaron,

Sorry, late night and I read your previous email too quickly.  Bryan is
correct, and I just tested it.  In the root pom, I added:


> 
> 
> org.apache.nifi
> nifi-api
> 1.10.0-SNAPSHOT
> 
> 
> org.apache.nifi
> nifi-mock
> 1.10.0-SNAPSHOT
> 
> 
> 
>

I also removed the version tag for nifi-mock (so that it inherits the
version specified in dependencyManagement) from the nifi-module (not
nifi-module-api, nifi-module-api-nar, or nifi-module-nar), and the build
succeeded.

On Mon, Jul 22, 2019 at 2:24 PM Bryan Bende  wrote:

> One more thing, make sure the  on the dependencies stays the
> same, meaning nifi-api should still be provided. You don't want to end
> up bundling the 1.10.0 nifi-api into your NARs.
>
> On Mon, Jul 22, 2019 at 2:21 PM Bryan Bende  wrote:
> >
> > I think you are correct that you would need to also bump nifi-mock to
> > 1.10.0-SNAPSHPOT.
> >
> > An easy way to make sure everything gets set to 1.10.0-SNAPSHOT is to
> > add a  block to the top-level pom of your
> > services project (the parent pom of the service, service api, and
> > NARs) and in the dep management declare the 1.10.0-SNAPSHOT for
> > nifi-api and nifi-mock.
> >
> > As a side note, there is obviously a slight risk here building against
> > 1.10.0 API and deploying to 1.9.2 since you could end up calling
> > something in your code that only exists in 1.10.0, but since it sounds
> > like you originally implemented against 1.9.2 then you are probably
> > fine. Just wanted to point this out.
> >
> > On Mon, Jul 22, 2019 at 2:04 PM Aaron Rich  wrote:
> > >
> > > Hi Jeff,
> > > Thanks for the information on the release process.
> > >
> > > For the error I'm getting, I started a new custom service with:
> > > mvn archetype:generate -DarchetypeGroupId=org.apache.nifi
> > > -DarchetypeArtifactId=nifi-service-bundle-archetype
> > > -DarchetypeVersion=1.9.2 -DnifiVersion=1.9.2
> > >
> > > This provides a shell test which I haven't updated.
> > >
> > > I updated the testservice-api-nar and the  testservice-nar poms like I
> did
> > > with the processor to include:
> > > 
> > > org.apache.nifi
> > > nifi-api
> > > 1.10.0-SNAPSHOT
> > > 
> > > 
> > >
> > > 
> > > 
> > > 
> > > org.apache.nifi
> > > nifi-nar-maven-plugin
> > > 1.3.1
> > > true
> > > 
> > > 
> > > 
> > >
> > > On the testService-api-nar I still get the warning of:
> > > [INFO] Generating documentation for NiFi extensions in the NAR...
> > > [INFO] Found NAR dependency of
> > > com.test.nifi:nifi-testService-api-nar:nar:1.0-SNAPSHOT:compile
> > > [INFO] Found NAR dependency of
> > > org.apache.nifi:nifi-standard-services-api-nar:nar:1.9.2:compile
> > > [INFO] Found NAR dependency of
> > > org.apache.nifi:nifi-jetty-bundle:nar:1.9.2:compile
> > > [INFO] Found a dependency on version 1.9.2 of NiFi API
> > > [WARNING] Could not generate extensions' documentation
> > > org.apache.maven.plugin.MojoExecutionException: Failed to create
> Extension
> > > Documentation
> > >
> > > When I was trying to trace back the 1.9.2 NiFi API dependency, I
> changed
> > > the testService to have nifi-api version 1.10. This is when the compile
> > > errors showed up.
> > >
> > > So at this point my question would be, for a custom service based on
> 1.9.2,
> > > which pom.xml files need to be changed, and with what changes, to have
> the
> > > service.nar file have the documentation to be able to be uploaded to
> nifi
> > > registry via cli? Seems like there is more then just moving nifi-api to
> > > version 1.10.
> > >
> > > Thank you again for the help.
> > >
> > > -Aaron
> > >
> > >
> > >
> > > On Mon, Jul 22, 2019 at 12:43 AM Jeff  wrote:
> > >
> > > > Aaron,
> > > >
> > > > New versions of NiFi are generally released after a member of the
> community
> > > > proposes a release and someone volunteers to perform the duties of
> the
> > > > release manager.  There is no predefined schedule or projected
> release
> > > > dates.  I think there are a few major feature JIRAs on which work is
> > &

Re: Required work to close out 1.10

[Jeff]

Another thought might be to get NiFi Registry builds on Java 11 working as
well for the NiFi Registry 0.5.0 release.  With the NiFi Java 11 build
compatibility close to being merged, it should be much more simple to get
NiFi Registry building on Java 11.  It would be good to release versions
for both projects to support Java 11 builds close together.

We could release NiFi 1.10.0 with Java 11 build compatibility, and then
release NiFi Registry 0.6.0 with Java 11 build compatibility, but I'm not
sure what other features would be included in NiFi Registry 0.6.0 other
than the Java 11 build compatibility.  I realize that might push back the
NiFi Registry 0.5.0 release timeframe back, probably weeks.

On Mon, Jul 29, 2019 at 3:29 PM Andy LoPresto  wrote:

> There are also a number of new features around encrypted repositories and
> encrypted configuration values that would be good to get in. Some have
> active PRs up right now, and others are close to being posted.
>
> Andy LoPresto
> alopre...@apache.org
> alopresto.apa...@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> > On Jul 29, 2019, at 12:27 PM, Bryan Bende  wrote:
> >
> > There are two really helpful bug fixes [1][2] related to version
> > controlling flows that I would like to see make it in to 1.10.0, but
> > they are dependent on releasing registry 0.5.0 first, since part of
> > the fixes are in the registry flow-diff code which NiFi depends on.
> >
> > So I was hoping to release registry 0.5.0 first, then nifi 1.10.0.
> >
> > If we took that approach I think there are still a few remaining items
> > before we can make a registry release.
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-6025
> > [2] https://issues.apache.org/jira/browse/NIFI-6314
> >
> > On Mon, Jul 29, 2019 at 3:12 PM Mike Thomsen 
> wrote:
> >>
> >> This thread is to just get a discussion started on what tickets need to
> be
> >> closed out before we should do 1.10.
> >>
> >> For graph there are two essential bug fixes:
> >>
> >> https://github.com/apache/nifi/pull/3571
> >> https://github.com/apache/nifi/pull/3572
> >>
> >> I think Jeff's Java 11 build support should also be included if
> possible:
> >>
> >> https://github.com/apache/nifi/pull/3404/
> >>
> >> If anyone needs some help on a review, feel free to ping me on GitHub.
> >>
> >> Mike
>
>

Re: Problems with build in non-english (german) environment

[Jeff]

Marc,

Your issue will most likely be resolved by NIFI-6529 [1], for which I
submitted PR 3639 [2].  If you get a chance, please check out PR 3639 and
run the tests as described in the PR.  You can update user.language and
user.region in the mvn command line to use your particular locale.

[1] https://issues.apache.org/jira/browse/NIFI-6529
[2] https://github.com/apache/nifi/pull/3639

On Wed, Aug 7, 2019 at 10:40 AM Arpad Boda  wrote:

> I think there are more of these:
> https://issues.apache.org/jira/browse/NIFI-5750
>
> On Wed, Aug 7, 2019 at 7:54 AM Marc Pellmann  wrote:
>
> > Hi Pierre,
> >
> > thank you for your reply! It should be reproduce-able with setting the
> > german locale in the pom:
> >
> > -Duser.language=de -Duser.country=DE
> > -Duser.region=DE
> >
> > Thanks,
> > Marc
> >
> > On Wed, Aug 7, 2019 at 1:21 PM Pierre Villard <
> pierre.villard...@gmail.com
> > >
> > wrote:
> >
> > > I filed a JIRA NIFI-6531 [1], and will look into it.
> > >
> > > [1] https://issues.apache.org/jira/browse/NIFI-6531
> > >
> > > Le mer. 7 août 2019 à 11:37, Pierre Villard <
> pierre.villard...@gmail.com
> > >
> > > a
> > > écrit :
> > >
> > > > Hi Marc,
> > > >
> > > > This should not happen as we are trying to have a build locale
> > agnostic.
> > > > If you can list the failing tests and/or file a JIRA, that would be
> > > > awesome. I'll definitely have a look on my side.
> > > >
> > > > The workaround you suggested about modifying the pom for the surefire
> > > > plugin is the best way to go AFAIK.
> > > >
> > > > Thanks,
> > > > Pierre
> > > >
> > > >
> > > > Le mer. 7 août 2019 à 11:05, Marc Pellmann  a
> > écrit
> > > :
> > > >
> > > >> Hi,
> > > >>
> > > >> I tried to build NiFi from source and had some problems with my
> german
> > > >> OS(X). Some tests did not passed because there have been errors
> with .
> > > vs
> > > >> ,
> > > >> in the tests.
> > > >>
> > > >> Example:
> > > >>
> > > >> [ERROR]
> testFormatDataSize(org.apache.nifi.processor.TestFormatUtils)
> > > >> Time
> > > >> elapsed: 0.004 s <<< FAILURE! org.junit.ComparisonFailure:
> > > >> expected:<10[.]4
> > > >> bytes> but was:<10[,]4 bytes> at
> > > >>
> > > >>
> > >
> >
> org.apache.nifi.processor.TestFormatUtils.testFormatDataSize(TestFormatUtils.java:91)
> > > >>
> > > >> Of course it works when I switch my OS to english/US, but that is
> not
> > > >> really a solution. To set this with system variables like
> > > >> JAVA_TOOL_OPTIONS
> > > >> with -Duser.language=en -Duser.country=US -Duser.region=US did not
> > > worked.
> > > >>
> > > >> The only solution was to set it in the pom.xml for the surefire
> plugin
> > > >> directly:
> > > >>
> > > >> -Duser.language=en -Duser.country=US
> > > >> -Duser.region=US
> > > >>
> > > >> Is there a different solution? Or if not - shouldn't this be the
> > > default?
> > > >>
> > > >> Thanks,
> > > >>
> > > >> Marc
> > > >>
> > > >
> > >
> >
>

Java 11 build support is live!

[Jeff]

Apache NiFi Developers,

Apache NiFi can now be built with Java 11!  Thanks to those that helped
with the review and committing of several PRs to get us to this point.

For all contributions going forward, developers, reviewers, and committers
need to make sure that PRs are verified by building with and running on
Java 8 and 11.

To assist in this process, the Linux-based Travis CI build has been updated
to perform an en_US locale-based build of NiFi using AdoptOpenJDK 11.0.4 in
addition to the three locale-based builds (en_US, fr_FR, ja_JP) on OpenJDK
8.

Apache NiFi still has a minimum requirement of Java 8 and as long as that
requirement exists the released convenience binaries will be built with
Java 8.

When building locally with Java 11, on all operating systems, a minimum JDK
version of 11.0.3 is required for a successful build.  If you are building
on OSX and use AdoptOpenJDK 11.0.4, there's an issue [4] [5] with native
libraries.  If you run into this issue, please downgrade your AdoptOpenJDK
11 installation to 11.0.3.  A updated release of AdoptOpenJDK 11.0.4 that
fixes the native library issues is forthcoming.  There is a comment [6] on
AdoptOpenJDK issue 1211 that describes the downgrade process.

Work still remains regarding Java 11.  Some of the components that NiFi
uses are not completely compatible with Java 11, and JIRAs will be created
or linked to the Java 11 parent JIRA [1] to track the remaining changes.

Several PRs not explicitly related to Java 11 were created and merged to
master ahead of NIFI-5176 [2] (PR 3404 [3]), and are listed as blockers on
NIFI-5176.  All blockers have been resolved, as well as NIFI-5176.

If you encounter an issue with the Java 11 build, or running the resulting
build on Java 11, please check NIFI-5174 to see if a JIRA for the issue has
already been filed, and create one if needed.

[1] https://issues.apache.org/jira/browse/NIFI-5174
[2] https://issues.apache.org/jira/browse/NIFI-5176
[3] https://github.com/apache/nifi/pull/3404
[4] https://github.com/AdoptOpenJDK/openjdk-build/issues/1206
[5] https://github.com/AdoptOpenJDK/openjdk-build/issues/1211
[6]
https://github.com/AdoptOpenJDK/openjdk-build/issues/1211#issuecomment-521392147

Re: Java 11 build support is live!

[Jeff]

Thanks Andy and Russell!

Another thing that might benefit the adoption/migration to Java 11 is some
additional information on various open/free JDK platforms that have
surfaced since Oracle changed the licensing.  Personally, most of the work
was done on OpenJDK 11.0.2, and then AdoptOpenJDK 11.0.3.  After the Java
11 PR was merged, I started looking at Zulu (from Azul).  From a community
perspective, I don't think we are interested in advocating for a specific
distribution of the JDK, and it would add a bunch of documentation work to
include information about all the known distributions.  There should be a
happy middle ground, though.

I'll be looking through our documentation and identifying areas for
improvement regarding building and running on Java 11.

On Thu, Aug 15, 2019 at 5:57 PM Russell Bateman 
wrote:

> Yes, at the risk of adding nothing more than a "me too," I nevertheless
> wish to add my thanks. Good job!
>
> On 8/15/19 12:04 PM, Jeff wrote:
>
> Apache NiFi Developers,
>
> Apache NiFi can now be built with Java 11!  Thanks to those that helped
> with the review and committing of several PRs to get us to this point.
>
> For all contributions going forward, developers, reviewers, and committers
> need to make sure that PRs are verified by building with and running on
> Java 8 and 11.
>
> To assist in this process, the Linux-based Travis CI build has been updated
> to perform an en_US locale-based build of NiFi using AdoptOpenJDK 11.0.4 in
> addition to the three locale-based builds (en_US, fr_FR, ja_JP) on OpenJDK
> 8.
>
> Apache NiFi still has a minimum requirement of Java 8 and as long as that
> requirement exists the released convenience binaries will be built with
> Java 8.
>
> When building locally with Java 11, on all operating systems, a minimum JDK
> version of 11.0.3 is required for a successful build.  If you are building
> on OSX and use AdoptOpenJDK 11.0.4, there's an issue [4] [5] with native
> libraries.  If you run into this issue, please downgrade your AdoptOpenJDK
> 11 installation to 11.0.3.  A updated release of AdoptOpenJDK 11.0.4 that
> fixes the native library issues is forthcoming.  There is a comment [6] on
> AdoptOpenJDK issue 1211 that describes the downgrade process.
>
> Work still remains regarding Java 11.  Some of the components that NiFi
> uses are not completely compatible with Java 11, and JIRAs will be created
> or linked to the Java 11 parent JIRA [1] to track the remaining changes.
>
> Several PRs not explicitly related to Java 11 were created and merged to
> master ahead of NIFI-5176 [2] (PR 3404 [3]), and are listed as blockers on
> NIFI-5176.  All blockers have been resolved, as well as NIFI-5176.
>
> If you encounter an issue with the Java 11 build, or running the resulting
> build on Java 11, please check NIFI-5174 to see if a JIRA for the issue has
> already been filed, and create one if needed.
>
> [1] https://issues.apache.org/jira/browse/NIFI-5174
> [2] https://issues.apache.org/jira/browse/NIFI-5176
> [3] https://github.com/apache/nifi/pull/3404
> [4] https://github.com/AdoptOpenJDK/openjdk-build/issues/1206
> [5] https://github.com/AdoptOpenJDK/openjdk-build/issues/1211
> [6]https://github.com/AdoptOpenJDK/openjdk-build/issues/1211#issuecomment-521392147
>
>
>

Re: [DISCUSS] Assembly size for 1.10

[Jeff]

How motivated could we be to do the reorganization of the NiFi repository
before the 1.10.0 release?  It sounds like we have a few paths to get the
resulting convenience binary down below the size limit.  If we don't do the
reorganization for this upcoming release, we should make it a top priority
for the following release.

On Wed, Aug 21, 2019 at 6:22 PM Mike Thomsen  wrote:

> Another factor on why that would be a good idea: I might soon have to pivot
> and do the R&D on adding dgraph support to graph bundle. So it's not
> altogether unlikely that it might need to be refactored to make room for
> other graph tech.
>
> On Wed, Aug 21, 2019 at 6:20 PM Mike Thomsen 
> wrote:
>
> > Go ahead and remove the whole graph bundle from the assembly. I would
> > recommend cutting a release of it separately and putting it up on
> GitHub's
> > releases listing if that's a possibility for us/INFRA w/ GitHub. Most of
> > our potential graph users are savvy enough that if add a few steps, I
> don't
> > see it causing any grief on them getting it stood up and giving us
> feedback.
> >
> > Might be a good idea also to add a "full-build" profile to the assembly
> so
> > that we can throw the whole kitchen sink into an unofficial build if we
> > build it ourselves for someone else.
> >
> > On Wed, Aug 21, 2019 at 3:09 PM Joe Witt  wrote:
> >
> >> Bryan
> >>
> >> I agree with all of that.  What does that get us to?
> >>
> >> Thanks
> >>
> >> On Wed, Aug 21, 2019 at 3:03 PM Bryan Bende  wrote:
> >>
> >> > I would vote to make nifi-flume-nar optional, and it looks like
> >> > nifi-other-graph-services-nar might be new since last release, so
> >> > since that is in the top 10 and not released yet, it might also be a
> >> > good candidate (not downplaying the usefulness of anything in that
> >> > NAR).
> >> >
> >> > I would also think we could consider the nifi-kafka-0-8-nar since
> >> > Kafka 0.8 is quite old at this point, and we already have other Kafka
> >> > NARs for 0.9, 0.10, 0.11, 1.0, and 2.0. Might even consider dropping a
> >> > few more versions from default assembly.
> >> >
> >> > On Wed, Aug 21, 2019 at 2:45 PM Aldrin Piri 
> wrote:
> >> > >
> >> > > Hi folks,
> >> > >
> >> > > Doing a recent PR review and build, it seems that master has amassed
> >> some
> >> > > additional size since our 1.9.2 release approaching 200MB.
> >> > >
> >> > > Unfortunately, this is problematic and needs to be addressed in
> >> advance
> >> > of
> >> > > our 1.10 release.  INFRA has been more than helpful making one off
> >> > > exceptions [1][2] for the larger assembly to get published to the
> ASF
> >> > > repository and its associated mirrors, but another release that is
> >> even
> >> > > larger is not something we can allow.  In a Linux environment, the
> >> master
> >> > > build reports in at 1575671276 which puts us over the hard limit
> >> > > highlighted in [2].
> >> > >
> >> > > We had a prior community discussion [3] about splitting the
> framework
> >> and
> >> > > extension repos and I am hoping to revive that discussion, in part.
> >> We
> >> > > certainly know what our longer term goals and ambitions are but
> need a
> >> > fix
> >> > > in the interim.  In the current state, we will not be able to make
> our
> >> > > convenience binaries available at the conclusion of the release
> >> process.
> >> > >
> >> > > At minimum we should evaluate which bundles are eligible to get
> >> treated
> >> > as
> >> > > optional dependencies and only enabled via profile, much like the
> work
> >> > that
> >> > > has occurred surrounding some of our other, hefty NARs. [4] A
> listing
> >> of
> >> > > the top 50 largest NARs, excluding framework and standard, is
> >> available
> >> > in
> >> > > a gist [5].  The nifi-media-nar looks to be a good initial candidate
> >> for
> >> > > exclusion.
> >> > >
> >> > > Thanks for your consideration!
> >> > >
> >> > > --aldrin
> >> > >
> >> > > [1] https://issues.apache.org/jira/browse/INFRA-11252
> >> > > [2] https://issues.apache.org/jira/browse/INFRA-15816
> >> > > [3]
> >> > >
> >> >
> >>
> https://lists.apache.org/thread.html/939a7630a2e32594cd10444e48b7a1321fd9ce51834d911a8c04b6a9@
> >> > 
> >> > > [4]
> >> > >
> >> >
> >>
> https://github.com/apache/nifi/blob/master/nifi-assembly/pom.xml#L807-L875
> >> > > [5] https://gist.github.com/apiri/4d9a02f9f6b46867b601956df83b6d8c
> >> >
> >>
> >
>

Re: [DISCUSS] Assembly size for 1.10

[Jeff]

Realistically we'll have to do the reorganization and removal of optional
NARs.  Regardless of the reorganization, the convenience binary has grown
too large.  Given that, it probably makes the most sense to pull out an
initial set of optional NARs from the convenience binary this release, and
implement the reorganization in the following release.

On Wed, Aug 21, 2019 at 10:06 PM Mike Thomsen 
wrote:

> My impression was that the reorganization initiative has enough steps that
> it might actually be wise for us to at least take on a chunk of them now so
> we don't end up with a release that suddenly feels to NiFi users the way
> Java 9 felt to Java 7 and 8 users.
>
> On Wed, Aug 21, 2019 at 9:59 PM Jeff  wrote:
>
> > How motivated could we be to do the reorganization of the NiFi repository
> > before the 1.10.0 release?  It sounds like we have a few paths to get the
> > resulting convenience binary down below the size limit.  If we don't do
> the
> > reorganization for this upcoming release, we should make it a top
> priority
> > for the following release.
> >
> > On Wed, Aug 21, 2019 at 6:22 PM Mike Thomsen 
> > wrote:
> >
> > > Another factor on why that would be a good idea: I might soon have to
> > pivot
> > > and do the R&D on adding dgraph support to graph bundle. So it's not
> > > altogether unlikely that it might need to be refactored to make room
> for
> > > other graph tech.
> > >
> > > On Wed, Aug 21, 2019 at 6:20 PM Mike Thomsen 
> > > wrote:
> > >
> > > > Go ahead and remove the whole graph bundle from the assembly. I would
> > > > recommend cutting a release of it separately and putting it up on
> > > GitHub's
> > > > releases listing if that's a possibility for us/INFRA w/ GitHub. Most
> > of
> > > > our potential graph users are savvy enough that if add a few steps, I
> > > don't
> > > > see it causing any grief on them getting it stood up and giving us
> > > feedback.
> > > >
> > > > Might be a good idea also to add a "full-build" profile to the
> assembly
> > > so
> > > > that we can throw the whole kitchen sink into an unofficial build if
> we
> > > > build it ourselves for someone else.
> > > >
> > > > On Wed, Aug 21, 2019 at 3:09 PM Joe Witt  wrote:
> > > >
> > > >> Bryan
> > > >>
> > > >> I agree with all of that.  What does that get us to?
> > > >>
> > > >> Thanks
> > > >>
> > > >> On Wed, Aug 21, 2019 at 3:03 PM Bryan Bende 
> wrote:
> > > >>
> > > >> > I would vote to make nifi-flume-nar optional, and it looks like
> > > >> > nifi-other-graph-services-nar might be new since last release, so
> > > >> > since that is in the top 10 and not released yet, it might also
> be a
> > > >> > good candidate (not downplaying the usefulness of anything in that
> > > >> > NAR).
> > > >> >
> > > >> > I would also think we could consider the nifi-kafka-0-8-nar since
> > > >> > Kafka 0.8 is quite old at this point, and we already have other
> > Kafka
> > > >> > NARs for 0.9, 0.10, 0.11, 1.0, and 2.0. Might even consider
> > dropping a
> > > >> > few more versions from default assembly.
> > > >> >
> > > >> > On Wed, Aug 21, 2019 at 2:45 PM Aldrin Piri 
> > > wrote:
> > > >> > >
> > > >> > > Hi folks,
> > > >> > >
> > > >> > > Doing a recent PR review and build, it seems that master has
> > amassed
> > > >> some
> > > >> > > additional size since our 1.9.2 release approaching 200MB.
> > > >> > >
> > > >> > > Unfortunately, this is problematic and needs to be addressed in
> > > >> advance
> > > >> > of
> > > >> > > our 1.10 release.  INFRA has been more than helpful making one
> off
> > > >> > > exceptions [1][2] for the larger assembly to get published to
> the
> > > ASF
> > > >> > > repository and its associated mirrors, but another release that
> is
> > > >> even
> > > >> > > larger is not something we can allow.  In a Linux environment,
> the
> > > >> master
> > > >> > > build reports i

Re: [discuss] approaching a NiFi 1.10.0 release

[Jeff]

I'd also like to get the PR [1] for NIFI-6275 merged before 1.10.0 is
released.  It's an update to ListHDFS so that the scheme and authority are
ignored when using the "Full Path" filter mode.

[1] https://github.com/apache/nifi/pull/3483

On Thu, Aug 29, 2019 at 10:04 PM Aldrin Piri  wrote:

> I created NIFI-6604 [1] to reduce assembly size and listed it as a Blocker
> for this release.  The issue has a link to the associated discussion thread
> and an initial PR.
>
> --aldrin
>
> [1] https://issues.apache.org/jira/browse/NIFI-6604
>
> On Thu, Aug 29, 2019 at 4:08 PM Matt Burgess  wrote:
>
> > Mike,
> >
> > I’ll review those two graph PRs tonight or tomorrow (if they’re still
> open
> > by then)
> >
> > > On Aug 29, 2019, at 3:43 PM, Mike Thomsen 
> > wrote:
> > >
> > > I have two open graph-related PR's that are really small and are needed
> > to
> > > close some bugs that will be bad for early adopters:
> > >
> > > https://github.com/apache/nifi/pull/3571
> > > https://github.com/apache/nifi/pull/3572
> > >
> > > If someone wants to review, that'd be great. Otherwise, I can merge
> them
> > in
> > > because I'm doing daily work on some graph stuff using a branch based
> on
> > > both patches.
> > >
> > > Thanks,
> > >
> > > Mike
> > >
> > >> On Thu, Aug 29, 2019 at 2:12 PM Joe Witt  wrote:
> > >>
> > >> We had another discuss on that recently and the intent is to drop a
> few
> > >> toys off the raft and update migration guidance.
> > >>
> > >> Thanks
> > >>
> > >>> On Thu, Aug 29, 2019 at 2:02 PM Jeremy Dyer 
> wrote:
> > >>>
> > >>> +1 looking forward to this.
> > >>>
> > >>> I recall seeing some issues about Apache Infra and the binary size.
> > Were
> > >>> all of those appropriate modules removed as discussed and the build
> > size
> > >>> will be small enough now?
> > >>>
> > >>>> On Thu, Aug 29, 2019 at 1:35 PM Bryan Bende 
> wrote:
> > >>>>
> > >>>> +1 Looking forward to getting parameters and Java 11 support out
> there
> > >>>> in a release.
> > >>>>
> > >>>>> On Thu, Aug 29, 2019 at 1:02 PM Joe Witt 
> wrote:
> > >>>>>
> > >>>>> Team,
> > >>>>>
> > >>>>> It looks like we're reaching a point in which it is time to close
> in
> > >> on
> > >>>>> 1.10.
> > >>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> https://issues.apache.org/jira/browse/NIFI-6595?jql=project%20%3D%20NIFI%20AND%20fixVersion%20%3D%201.10.0
> > >>>>>
> > >>>>> There are 250+ issues in there nearly 240 of which are already
> > >>>> resolved.  I
> > >>>>> haven't gone through yet for fully analyzing but the awesome Java
> 11
> > >>> work
> > >>>>> Jeff Storck and others has done means we can now build on Java 11
> as
> > >>> well
> > >>>>> as run on it.  There is a ton of great work on parameters which
> will
> > >>> make
> > >>>>> managing flows at scale much easier.  We will need to drop some
> nars
> > >>>> which
> > >>>>> means we have some migration guidance to update.
> > >>>>>
> > >>>>> I'd like to volunteer as RM but if there are any other takers
> please
> > >>> let
> > >>>> me
> > >>>>> know.
> > >>>>>
> > >>>>> As we close in on releases this tends to create a lot of urgency to
> > >>>> quickly
> > >>>>> try to get new things in.  I will try to manage this well.  As
> always
> > >>> we
> > >>>>> can do another release.  There is no timeline pushing us to have
> such
> > >>>> gaps
> > >>>>> between releases...we can always do another feature release.  That
> > >>> said,
> > >>>> PR
> > >>>>> review bandwidth is at a super premium.  We receive a lot more PRs
> > >> than
> > >>>> we
> > >>>>> do receive reviews.  We'll have to work this as the PR depth grows.
> > >>>>>
> > >>>>> Thanks
> > >>>>> Joe
> > >>>>
> > >>>
> > >>
> >
>

Re: [EXT] Re: [VOTE] Create NiFi Standard Libraries sub-project

[Jeff]

+1 Create NiFi Standard Libraries (binding)

On Wed, Sep 4, 2019 at 12:19 AM Peter Wicks (pwicks) 
wrote:

> +1, binding
>
> -Original Message-
> From: Kevin Doran 
> Sent: Tuesday, September 3, 2019 7:12 PM
> To: dev@nifi.apache.org; dev@nifi.apache.org
> Subject: [EXT] Re: [VOTE] Create NiFi Standard Libraries sub-project
>
> +1, binding
>
>
> 
> From: Tony Kurc 
> Sent: Tuesday, September 3, 2019 8:33 PM
> To: dev@nifi.apache.org
> Subject: Re: [VOTE] Create NiFi Standard Libraries sub-project
>
> +1 (binding)
>
> On Wed, Sep 4, 2019 at 12:29 AM Aldrin Piri  wrote:
>
> > +1, binding
> >
> > On Tue, Sep 3, 2019 at 19:46 Yolanda Davis 
> > wrote:
> >
> > > +1 Create NiFi Standard Libraries (binding)
> > >
> > > On Tue, Sep 3, 2019 at 7:03 PM Koji Kawamura
> > > 
> > > wrote:
> > >
> > > > +1 Create NiFi Standard Libraries (binding)
> > > >
> > > > On Wed, Sep 4, 2019 at 7:25 AM Mike Thomsen
> > > > 
> > > > wrote:
> > > > >
> > > > > +1 binding
> > > > >
> > > > > On Tue, Sep 3, 2019 at 5:33 PM Andy LoPresto
> > > > > 
> > > > wrote:
> > > > >
> > > > > > +1, create NiFi Standard Libraries (binding)
> > > > > >
> > > > > > Andy LoPresto
> > > > > > alopre...@apache.org
> > > > > > alopresto.apa...@gmail.com
> > > > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D
> > > > > > EF69
> > > > > >
> > > > > > > On Sep 3, 2019, at 2:16 PM, Bryan Bende 
> > wrote:
> > > > > > >
> > > > > > > All,
> > > > > > >
> > > > > > > In a previous thread there was a plan discussed to
> > > > > > > restructure
> > some
> > > > of
> > > > > > > the repositories in order to address several different
> > > > > > > issues,
> > such
> > > > as
> > > > > > > build time, reusability of code, and eventually separating
> > > > > > > how
> > the
> > > > > > > framework and extensions are released [1][2].
> > > > > > >
> > > > > > > The overall plan requires many steps to get there, so I'd
> > > > > > > like to propose starting with a small actionable step - the
> > > > > > > creation of a
> > > new
> > > > > > > sub-project called NiFi Standard Libraries (formerly
> > > > > > > referred to
> > as
> > > > > > > nifi-commons).
> > > > > > >
> > > > > > > Project Name: Apache NiFi Standard Libraries Git Repository:
> > > > > > > nifi-standard-libraries
> > > > > > > JIRA: NIFILIBS
> > > > > > >
> > > > > > > Description:
> > > > > > >
> > > > > > > A collection of standard implementations used across the
> > > > > > > NiFi
> > > > ecosystem.
> > > > > > >
> > > > > > > Candidate Libraries:
> > > > > > >
> > > > > > > In general, each library may consist of multiple Maven
> > > > > > > modules,
> > and
> > > > > > > should be independent from the rest of the ecosystem, and
> > > > > > > from
> > > other
> > > > > > > libraries within NiFi Standard Libraries.
> > > > > > >
> > > > > > > In addition, each library may make it's own decision about
> > whether
> > > it
> > > > > > > is considered a public facing extension point/API, or an
> > > > > > > internal library that may be changed at any time. This
> > > > > > > should be
> > documented
> > > in
> > > > > > > a README at the root of each library, such as
> > > > > > > nifi-standard-libraries/nifi-xyz/README.
> > > > > > >
> > > > > > > An initial library that has been discussed was referred to
> > > > > > > as 'nifi-security' and would centralize much of the security
> > > > > > > related
> > > > code
> > > > > > > shared by NiFi and NiFi Registry, such as shared security
> > > > > > > APIs,
> > and
> > > > > > > implementations for various providers, such as
> LDAP/Kerberos/etc.
> > > > > > >
> > > > > > > A second candidate library would be an optimistic-locking
> > > > > > > library based on NiFi's revision concept. Currently this has
> > > > > > > been created inside nifi-registry for now [3], but could be
> > > > > > > moved as soon as nifi-standard-libraries exists.
> > > > > > >
> > > > > > > (This list does not have to be final in order to decide if
> > > > > > > we are creating NiFi Standard Libraries or not)
> > > > > > >
> > > > > > > Integration & Usage:
> > > > > > >
> > > > > > > Once NiFi Standard Libraries is created, the community can
> > > > > > > start creating and/or moving code there and perform releases
> > > > > > > as
> > > necessary.
> > > > A
> > > > > > > release will consist of the standard Apache source release,
> > > > > > > plus artifacts released to Maven central. The community can
> > > > > > > then
> > decide
> > > > > > > when it is appropriate to integrate these released libraries
> > > > > > > into
> > > one
> > > > > > > of our downstream projects.
> > > > > > >
> > > > > > > For example, if we create a nifi-security library in
> > > > > > > nifi-standard-libraries, we can release that whenever we
> > > > > > > decide,
> > > but
> > > > > > > we may not integrate it into NiFi or NiFi Registry until it
> > > > > > > makes sense for a given release of those projects.
> > > > > > >
> > > > > > 

Re: [ANNOUNCE] New Apache NiFi Committer Rob Fellows

[Jeff]

Congrats, Rob!

On Wed, Sep 25, 2019 at 11:12 AM Matt Gilman 
wrote:

> Welcome and congrats Rob!
>
> On Wed, Sep 25, 2019 at 10:52 AM Yolanda Davis 
> wrote:
>
> > Congratulations Rob!
> >
> >
> > > On Sep 25, 2019, at 10:09 AM, Marc Parisi  wrote:
> > >
> > > Congrats Rob!
> > >
> > >> On Wed, Sep 25, 2019 at 9:59 AM Rob Moran  wrote:
> > >>
> > >> Thanks for all your work, Rob -- congrats!
> > >>
> > >> On Wed, Sep 25, 2019 at 9:29 AM Andrew Lim <
> andrewlim.apa...@gmail.com>
> > >> wrote:
> > >>
> > >>> Congrats Rob!
> > >>>
> > >>>
> >  On Sep 24, 2019, at 7:56 PM, Tony Kurc  wrote:
> > 
> >  Apache NiFi community,
> >  On behalf of the Apache NiFI PMC, I am very pleased to announce that
> > >> Rob
> >  has accepted the PMC's invitation to become a committer on the
> Apache
> > >>> NiFi
> >  project. We greatly appreciate all of Rob's hard work and generous
> >  contributions to the project. We look forward to his continued
> > >>> involvement
> >  in the project.
> > 
> >  What stood out with Rob are his regular code contributions and
> reviews
> > >> on
> >  many parts of the project to include NiFi, NiFi FDS, and NiFi
> Registry
> >  since early this year. Additionally, he's been doing the
> >  not-always-glamorous work of helping verify releases, which was a
> huge
> >  assist in getting NiFi 1.9.1, NiFi Registry 0.4.0 and 0.5.0, and
> NiFi
> > >> FDS
> >  0.2.0 out to the community.
> > 
> >  Welcome and congratulations!
> >  Tony
> > >>>
> > >>>
> > >>
> >
>

Re: NIFI-6597 Build with apache-rat-plugin failure

[Jeff]

This looks like it might be caused by unresolved merge conflicts.

On Tue, Sep 24, 2019 at 1:49 PM Bryan Bende  wrote:

> Did you run with -Pcontrib-check when building locally?
>
> The issue from the Travis logs is:
>
> [WARNING] Files with unapproved licenses:
>
> /home/travis/build/apache/nifi/nifi-framework-api/src/main/java/org/apache/nifi/diagnostics/DiagnosticsDump.java.orig
>
> Looking at the PR, there are lot of files that are somehow renamed to .orig
>
> On Tue, Sep 24, 2019 at 1:36 PM Sunny Zhang
>  wrote:
> >
> > Hi my PR NIFI-6597 failed on the auto build test with problems of
> apache-rat-plugin. But When I build it on my local machine there's no error.
> >
> https://travis-ci.org/apache/nifi/builds/588696226?utm_source=github_status&utm_medium=notification
> >
> > Not sure what happened.. Could anyone help me on this?
> >
> > Thanks!!!
> > Sunny
> >
> > [INFO]
> 
> > 5327[INFO] BUILD FAILURE
> > 5328[INFO]
> 
> > 5329[INFO] Total time:  01:34 min (Wall Clock)
> > 5330[INFO] Finished at: 2019-09-23T23:18:42Z
> > 5331[INFO]
> 
> > 5332[ERROR] Failed to execute goal
> org.apache.rat:apache-rat-plugin:0.12:check (default) on project
> nifi-framework-api: Too many files with unapproved license: 1 See RAT
> report in: /home/travis/build/apache/nifi/nifi-framework-api/target/rat.txt
> -> [Help 1]
> > 5333[ERROR]
> > 5334[ERROR] To see the full stack trace of the errors, re-run Maven with
> the -e switch.
> > 5335[ERROR] Re-run Maven using the -X switch to enable full debug
> logging.
> > 5336[ERROR]
> > 5337[ERROR] For more information about the errors and possible
> solutions, please read the following articles:
> > 5338[ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> > 5339[ERROR]
> > 5340[ERROR] After correcting the problems, you can resume the build with
> the command
> > 5341[ERROR]   mvn  -rf :nifi-framework-api
> > 5342The command "mvn clean install -V -T 1C -pl `find . -type d \( -name
> "*-nar" -or -name "*-assembly" \) -and -not -name "*api-nar" -and -not
> -path "*/target/*" -and -not -name "*__*" -printf "!./%P,"`
> -Pcontrib-check,include-grpc -Ddir-only
> -Dmaven.surefire.arguments="-Duser.language=en -Duser.region=US" | grep -v
> -F -f .travis-output-filters && test ${PIPESTATUS[0]} -eq 0" exited with 1.
> > before_cache
> > 53430.05s$ rm -rf $HOME/.m2/repository/org/apache/nifi/
> > cache.2
> > 5344store build cache
> > 53450.01s89.55schanges detected (content changed, file is created, or
> file is
> deleted):\n/home/travis/.m2/repository/io/netty/netty-all/4.1.29.Final/netty-all-4.1.29.Final.jar
> >
> 5346/home/travis/.m2/repository/io/netty/netty-all/4.1.29.Final/netty-all-4.1.29.Final.jar.sha1
> >
> 5347/home/travis/.m2/repository/io/netty/netty-all/4.1.29.Final/netty-all-4.1.29.Final.pom
> >
> 5348/home/travis/.m2/repository/io/netty/netty-all/4.1.29.Final/netty-all-4.1.29.Final.pom.sha1
> >
> 5349/home/travis/.m2/repository/io/netty/netty-all/4.1.29.Final/_remote.repositories
> >
> 5350/home/travis/.m2/repository/io/netty/netty-parent/4.1.29.Final/netty-parent-4.1.29.Final.pom
> >
> 5351/home/travis/.m2/repository/io/netty/netty-parent/4.1.29.Final/netty-parent-4.1.29.Final.pom.sha1
> >
> 5352/home/travis/.m2/repository/io/netty/netty-parent/4.1.29.Final/_remote.repositories
> >
> 5353/home/travis/.m2/repository/org/apache/curator/curator-test/4.2.0/curator-test-4.2.0.jar
> >
> 5354/home/travis/.m2/repository/org/apache/curator/curator-test/4.2.0/curator-test-4.2.0.jar.sha1
> >
> 5355/home/travis/.m2/repository/org/apache/curator/curator-test/4.2.0/curator-test-4.2.0.pom
> > 5356/home/travis/.\n...
> > 5357changes detected, packing new archive
> > 5358uploading
> PR.3676/cache-linux-xenial-e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855--jdk-openjdk8.tgz
> > 5359cache uploaded
> > 5360
> > 5361
> > 5362Done. Your build exited with 1.
> >
> >
> > But when I build it I don't have any error:
> > [INFO]
> 
> > [INFO] BUILD SUCCESS
> > [INFO]
> 
> > [INFO] Total time:  14:32 min (Wall Clock)
> > [INFO] Finished at: 2019-09-24T10:08:29-07:00
> > [INFO]
> 
> > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option
> MaxPermSize=256m; support was removed in 8.0
> >
> >
> >
>

Re: [discuss] approaching a NiFi 1.10.0 release

[Jeff]

Thanks Joe!  I should have an update/resolution for NIFI-6275 tomorrow.

On Mon, Sep 30, 2019 at 11:48 PM Joe Witt  wrote:

> Team,
>
> We are pretty much there now it looks like for a 1.10.0.  In scanning
> through this thread it looks like the ListHDFS ask from Jeff is perhaps the
> only remaining bit.
>
> I'll probably create the RC branch soon so folks can keep moving on master
> and I'll try to pull in other things that look ready/reviewed/etc.. that
> might work.
>
> If you look here
> https://issues.apache.org/jira/projects/NIFI/versions/12344993 we already
> have a *TON* of features/fixes/improvements loaded up on this release.
>
> My goal is to initiate the RC processes this week.
>
> Huge thanks to the many many folks that contributed to getting us this far
> in this release.
>
> Thanks!
> Joe
>
>
>
> On Fri, Aug 30, 2019 at 4:21 AM Pierre Villard <
> pierre.villard...@gmail.com>
> wrote:
>
> > +1 for getting a 1.10.0 release soon - a lot of great things has been
> > added.
> >
> > If I may ask, it would be great to have a +1 from a committer on
> NIFI-6159
> > [1].
> > The PR has been reviewed by turcsanyip but it would need a final go from
> a
> > committer.
> > However, I do understand that other PRs mentioned in this thread are more
> > important (bug related).
> >
> > [1] https://github.com/apache/nifi/pull/3394
> >
> > Thanks,
> > Pierre
> >
> > Le ven. 30 août 2019 à 08:21, Jeff  a écrit :
> >
> > > I'd also like to get the PR [1] for NIFI-6275 merged before 1.10.0 is
> > > released.  It's an update to ListHDFS so that the scheme and authority
> > are
> > > ignored when using the "Full Path" filter mode.
> > >
> > > [1] https://github.com/apache/nifi/pull/3483
> > >
> > > On Thu, Aug 29, 2019 at 10:04 PM Aldrin Piri 
> > wrote:
> > >
> > > > I created NIFI-6604 [1] to reduce assembly size and listed it as a
> > > Blocker
> > > > for this release.  The issue has a link to the associated discussion
> > > thread
> > > > and an initial PR.
> > > >
> > > > --aldrin
> > > >
> > > > [1] https://issues.apache.org/jira/browse/NIFI-6604
> > > >
> > > > On Thu, Aug 29, 2019 at 4:08 PM Matt Burgess 
> > > wrote:
> > > >
> > > > > Mike,
> > > > >
> > > > > I’ll review those two graph PRs tonight or tomorrow (if they’re
> still
> > > > open
> > > > > by then)
> > > > >
> > > > > > On Aug 29, 2019, at 3:43 PM, Mike Thomsen <
> mikerthom...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > I have two open graph-related PR's that are really small and are
> > > needed
> > > > > to
> > > > > > close some bugs that will be bad for early adopters:
> > > > > >
> > > > > > https://github.com/apache/nifi/pull/3571
> > > > > > https://github.com/apache/nifi/pull/3572
> > > > > >
> > > > > > If someone wants to review, that'd be great. Otherwise, I can
> merge
> > > > them
> > > > > in
> > > > > > because I'm doing daily work on some graph stuff using a branch
> > based
> > > > on
> > > > > > both patches.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Mike
> > > > > >
> > > > > >> On Thu, Aug 29, 2019 at 2:12 PM Joe Witt 
> > > wrote:
> > > > > >>
> > > > > >> We had another discuss on that recently and the intent is to
> drop
> > a
> > > > few
> > > > > >> toys off the raft and update migration guidance.
> > > > > >>
> > > > > >> Thanks
> > > > > >>
> > > > > >>> On Thu, Aug 29, 2019 at 2:02 PM Jeremy Dyer 
> > > > wrote:
> > > > > >>>
> > > > > >>> +1 looking forward to this.
> > > > > >>>
> > > > > >>> I recall seeing some issues about Apache Infra and the binary
> > size.
> > > > > Were
> > > > > >>> all of those appropriate modules removed as discussed and the
> > build
> > > > > size
> 

Re: October 2019 board report

[Jeff]

Joe,

Thanks for submitting the report!  I noticed in the Project Activity
section, Apache NiFi's next release candidate is specified for version
0.10.0, instead of 1.10.0.

On Wed, Oct 2, 2019 at 12:30 AM Andy LoPresto 
wrote:

> Joe,
>
> Looks great. Thanks for putting this together. One comment: the commuter
> to PMC ratio is listed as 3:2 (44 to 30). I think the 44 number includes
> all PMC members as well, so I would present the ratio as 1:2. But perhaps
> I’m misunderstanding the point of that distinction.
>
> Andy LoPresto
> alopre...@apache.org
> alopresto.apa...@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> > On Oct 1, 2019, at 19:00, Joe Witt  wrote:
> >
> > Team,
> >
> > Here is the board report I've submitted for us this October.  Great
> > progress!
> >
> > For those familiar with the format we've typically used this is a bit
> > different. I used a new tool that helps structure and capture data for
> the
> > report a bit differently than the previous approach.  We'll see what the
> > feedback from the board is.
> >
> > Thanks
> > Joe
> >
> > -
> > ## Description:
> > The mission of NiFi is the creation and maintenance of software related
> to
> > providing an easy to use, powerful, and reliable system to process and
> > distribute data.
> >
> > Apache NiFi MiNiFi is an edge data collection agent built to seamlessly
> > integrate with and leverage the command and control of NiFi. There are
> both
> > Java and C++ implementations.
> >
> > Apache NiFi Registry is a centralized registry for key configuration
> items
> > including flow versions, assets, and extensions for Apache NiFi and
> Apache
> > MiNiFi.
> >
> > Apache NiFi Nar Maven Plugin is a release artifact used for supporting
> the
> > NiFi classloader isolation model.
> >
> > Apache NiFi Flow Design System is a theme-able set of high quality UI
> > components and utilities for use across the various Apache NiFi web
> > applications in order to provide a more consistent user experience.
> > ## Issues:
> > There are no issues requiring board attention at this time.
> >
> > ## Membership Data:
> > Apache NiFi was founded 2015-07-14 (4 years ago)
> > There are currently 44 committers and 30 PMC members in this project.
> > The Committer-to-PMC ratio is roughly 3:2.
> >
> > Community changes, past quarter:
> > - No new PMC members. Last addition was Peter Wicks on 2019-05-29.
> > - Rob Fellows was added as committer on 2019-09-24
> >
> > ## Project Activity:
> > Released Apache NiFi Registry 0.5.0 providing better proxy permissions,
> > support for Apache Knox, and the ability to make buckets accessible by
> > anonymous users.
> >
> > Apache NiFi 0.10.0 will have a release candidate likely by the time of
> this
> > October board report.  It includes more than 330+ JIRAs.  This brings
> > powerful
> > features such as sourcing extensions from the latest NiFi Registry at
> > runtime,
> > far better model for paramaterized version controlled flows, Java 11
> > compatibility, back pressure prediction and more.
> >
> > The release for MiNiFi CPP 0.7.0 is closing in.  There are nearly 100
> closed
> > JIRAs including many bug fixes and new features such as ensuring all
> inbound
> > sockets are TLSv1.2 or newer, cron driven scheduling, support Kerberized
> > connections to Kafka, support for packages with Python scripts, and
> ability
> > to
> > capture individual frames from an RTSP camera stream.
> > ## Community Health:
> > Community health continues to be a strong point with a healthy committer
> and
> > PMC pipeline as well as high activity from current PMC members. We see
> > numerous release lines across the mentioned parts of the Apache NiFi
> project
> > all with very active development across features, bug fixes, and
> > improvements.
> > Our mailing list continues to be highly active and our slack room is also
> > super active.
> >
> > As reported in the previous quarter we had 394 users in our slack room
> but
> > now
> > there are 523 and growing.  This is a great sign of the interest in the
> > community and the engagement level we see across PMC, committer, and
> > pipeline
> > for future committers.
> >
> > We continue to see highly active use and commentary related to Apache
> NiFi
> > in
> > social media, meetups, blogs, and conferences in various parts of the
> > world. A
> > NiFi PMC member presented on NiFi at the recent ApacheCon in Vegas. A
> recent
> > tweet reads "Work with ApacheNiFi is very fast and productive.  Counting
> now
> > +290 processors to build both batch and streaming ETL data pipelines for
> any
> > type of data source and destination you want." Another reads "ApacheNiFi
> > thanks for the usual level of nonsensical obfuscated badly described
> prose
> > on
> > your homepage trying to describe your framework/product. As usual
> > obfuscation
> > is your the forte of the Apache Foundation..." Fortunately patches are
> > welcome.
> >
> > This quarter we're seeing high

Re: [discuss] approaching a NiFi 1.10.0 release

[Jeff]

Joe,

NIFI-6275 [1] has been resolved, and PR 3483 [2] has been merged to master.

[1] https://issues.apache.org/jira/browse/NIFI-6275
[2] https://github.com/apache/nifi/pull/3483

On Tue, Oct 1, 2019 at 12:30 AM Jeff  wrote:

> Thanks Joe!  I should have an update/resolution for NIFI-6275 tomorrow.
>
> On Mon, Sep 30, 2019 at 11:48 PM Joe Witt  wrote:
>
>> Team,
>>
>> We are pretty much there now it looks like for a 1.10.0.  In scanning
>> through this thread it looks like the ListHDFS ask from Jeff is perhaps
>> the
>> only remaining bit.
>>
>> I'll probably create the RC branch soon so folks can keep moving on master
>> and I'll try to pull in other things that look ready/reviewed/etc.. that
>> might work.
>>
>> If you look here
>> https://issues.apache.org/jira/projects/NIFI/versions/12344993 we already
>> have a *TON* of features/fixes/improvements loaded up on this release.
>>
>> My goal is to initiate the RC processes this week.
>>
>> Huge thanks to the many many folks that contributed to getting us this far
>> in this release.
>>
>> Thanks!
>> Joe
>>
>>
>>
>> On Fri, Aug 30, 2019 at 4:21 AM Pierre Villard <
>> pierre.villard...@gmail.com>
>> wrote:
>>
>> > +1 for getting a 1.10.0 release soon - a lot of great things has been
>> > added.
>> >
>> > If I may ask, it would be great to have a +1 from a committer on
>> NIFI-6159
>> > [1].
>> > The PR has been reviewed by turcsanyip but it would need a final go
>> from a
>> > committer.
>> > However, I do understand that other PRs mentioned in this thread are
>> more
>> > important (bug related).
>> >
>> > [1] https://github.com/apache/nifi/pull/3394
>> >
>> > Thanks,
>> > Pierre
>> >
>> > Le ven. 30 août 2019 à 08:21, Jeff  a écrit :
>> >
>> > > I'd also like to get the PR [1] for NIFI-6275 merged before 1.10.0 is
>> > > released.  It's an update to ListHDFS so that the scheme and authority
>> > are
>> > > ignored when using the "Full Path" filter mode.
>> > >
>> > > [1] https://github.com/apache/nifi/pull/3483
>> > >
>> > > On Thu, Aug 29, 2019 at 10:04 PM Aldrin Piri 
>> > wrote:
>> > >
>> > > > I created NIFI-6604 [1] to reduce assembly size and listed it as a
>> > > Blocker
>> > > > for this release.  The issue has a link to the associated discussion
>> > > thread
>> > > > and an initial PR.
>> > > >
>> > > > --aldrin
>> > > >
>> > > > [1] https://issues.apache.org/jira/browse/NIFI-6604
>> > > >
>> > > > On Thu, Aug 29, 2019 at 4:08 PM Matt Burgess 
>> > > wrote:
>> > > >
>> > > > > Mike,
>> > > > >
>> > > > > I’ll review those two graph PRs tonight or tomorrow (if they’re
>> still
>> > > > open
>> > > > > by then)
>> > > > >
>> > > > > > On Aug 29, 2019, at 3:43 PM, Mike Thomsen <
>> mikerthom...@gmail.com>
>> > > > > wrote:
>> > > > > >
>> > > > > > I have two open graph-related PR's that are really small and are
>> > > needed
>> > > > > to
>> > > > > > close some bugs that will be bad for early adopters:
>> > > > > >
>> > > > > > https://github.com/apache/nifi/pull/3571
>> > > > > > https://github.com/apache/nifi/pull/3572
>> > > > > >
>> > > > > > If someone wants to review, that'd be great. Otherwise, I can
>> merge
>> > > > them
>> > > > > in
>> > > > > > because I'm doing daily work on some graph stuff using a branch
>> > based
>> > > > on
>> > > > > > both patches.
>> > > > > >
>> > > > > > Thanks,
>> > > > > >
>> > > > > > Mike
>> > > > > >
>> > > > > >> On Thu, Aug 29, 2019 at 2:12 PM Joe Witt 
>> > > wrote:
>> > > > > >>
>> > > > > >> We had another discuss on that recently and the intent is to
>> drop
>> > a
>> > > > few
>> > > > > >> toy

Re: [EXT] [ANNOUNCE] New Apache NiFi Committer Kotaro Terada

[Jeff]

Welcome, Kotaro!

On Fri, Oct 25, 2019 at 10:58 AM Koji Kawamura 
wrote:

> Congratulations Kotaro! I'm so glad to see another Japanese NiFi committer!
>
> On Fri, Oct 25, 2019 at 2:24 AM Pierre Villard
>  wrote:
> >
> > Congrats Kotaro!
> >
> > Le ven. 25 oct. 2019 à 10:19, koter...@yahoo-corp.jp <
> koter...@yahoo-corp.jp>
> > a écrit :
> >
> > > Thank you Aldrin and everyone!
> > > Looking forward to continuing working with you and the community!
> > >
> > > Thanks,
> > > Kotaro
> > >
> > > 
> > > From: Peter Wicks (pwicks) 
> > > Sent: Friday, October 25, 2019 11:44
> > > To: dev@nifi.apache.org 
> > > Subject: RE: [EXT] [ANNOUNCE] New Apache NiFi Committer Kotaro Terada
> > >
> > > おめでとう寺田さん！
> > >
> > > -Original Message-
> > > From: Aldrin Piri 
> > > Sent: Thursday, October 24, 2019 7:50 AM
> > > To: dev 
> > > Subject: [EXT] [ANNOUNCE] New Apache NiFi Committer Kotaro Terada
> > >
> > > Apache NiFi community,
> > >
> > > On behalf of the Apache NiFI PMC, I am very pleased to announce that
> Kotaro
> > > has accepted the PMC's invitation to become a committer on the Apache
> NiFi
> > > project. We greatly appreciate all of Kotaro's hard work and generous
> > > contributions to the project. We look forward to continued involvement
> > > in the project.
> > >
> > > Kotaro contributed to a breadth of areas in both NiFi and Registry as
> well
> > > as
> > > being a regular reviewer of our releases. Kotaro's communication in
> Jira
> > > issues
> > > and responsiveness to the review processes highlighted great
> collaboration
> > > and
> > > embodied our community goals for the project.
> > >
> > > Welcome and congratulations!
> > >
> > > --ap
> > >
>

Re: [ANNOUNCE] New Apache NiFi Committer Dániel Bakai

[Jeff]

Congrats and welcome, Dániel!

On Sat, Oct 26, 2019 at 8:39 AM Marc Parisi  wrote:

> Congrats!
>
> On Fri, Oct 25, 2019, 11:51 PM Sivaprasanna 
> wrote:
>
> > Congratulations, Daniel. All the very best.
> >
> > On Sat, 26 Oct 2019 at 6:01 AM, Tony Kurc  wrote:
> >
> > > Congratulations Dániel!
> > >
> > > On Fri, Oct 25, 2019, 8:21 PM Otto Fowler 
> > wrote:
> > >
> > > > std::cout << “Congratulations” << std::endl;
> > > >
> > > >
> > > >
> > > >
> > > > On October 25, 2019 at 12:38:20, Aldrin Piri (ald...@apache.org)
> > wrote:
> > > >
> > > > Apache NiFi community,
> > > >
> > > > On behalf of the Apache NiFI PMC, I am very pleased to announce that
> > > Dániel
> > > > has accepted the PMC's invitation to become a committer on the Apache
> > > NiFi
> > > > project. We greatly appreciate all of Dániel's hard work and generous
> > > > contributions to the project. We look forward to continued
> involvement
> > > > in the project.
> > > >
> > > > Dániel has provided numerous contributions to the MiNiFi C++
> codebase,
> > > > discovering and providing fixes for bugs, new functionality, and
> > > improving
> > > > build processes. Dániel is also a staple in review processes and
> > > > approaches each interaction with great communication and
> > professionalism.
> > > >
> > > > Welcome and congratulations!
> > > > AP
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Apache NiFi Committer Peter Turcsanyi

[Jeff]

Congrats and welcome, Peter!

On Tue, Oct 29, 2019 at 4:44 PM Peter Turcsanyi 
wrote:

> Thank you everyone!
>
> Peter
>
> On Mon, Oct 28, 2019 at 6:06 PM Andy LoPresto 
> wrote:
>
> > Congratulations Peter. Looking forward to your continued contributions.
> >
> > Andy LoPresto
> > alopre...@apache.org
> > alopresto.apa...@gmail.com
> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >
> > > On Oct 28, 2019, at 9:39 AM, Matt Burgess 
> wrote:
> > >
> > > Congratulations Peter!
> > >
> > > On Sun, Oct 27, 2019 at 9:05 PM Aldrin Piri 
> > wrote:
> > >>
> > >> Apache NiFi community,
> > >>
> > >> On behalf of the Apache NiFI PMC, I am very pleased to announce that
> > Peter
> > >> has accepted the PMC's invitation to become a committer on the Apache
> > NiFi
> > >> project. We greatly appreciate all of Peter's hard work and generous
> > >> contributions to the project. We look forward to continued involvement
> > >> in the project.
> > >>
> > >> Peter has provided several new extensions and improvements enhancing
> > NiFi's
> > >> interoperability with cloud services.  He has also found and remedied
> > >> several bugs, is a regular participant in code reviews and an active
> > >> presence on the mailing lists helping out the community.
> > >>
> > >> Welcome and congratulations!
> >
> >
>

Re: IntelliJ, language level and jigaw profile

[Jeff]

Hi Mark,

IntelliJ IDEA seems to implicitly activate the "jigsaw" profile with a
"light" checkmark if it thinks you have Java 11 installed versus a "bold"
checkmark if you've clicked that profile to be enabled explicitly.  Make
sure that profile is unchecked, and make IDEA rebuild.  That should get you
back to having your classes compiled with Java 8.

On Wed, Jan 29, 2020 at 4:23 PM Mark Bean  wrote:

> I'm having trouble with IntelliJ setting the proper language level. Has
> anyone else seen the following behavior?
>
> IntelliJ 2019.3.2 (Community Edition)
> installed from ideaIC-2019.3.2-no-jbr.tar.gz
>
> File > Project Structure > Platform Settings > SDKs
>   - Only 1.8 is loaded
> File > Project Structure > Project Settings > Project
>   - Project SDK = 1.8
>   - Project language level = 8 - Lambdas, type annotations etc.
> File > Project Structure > Project Settings > Modules
>   - Every module lists "Language level" as "11 - Local variable syntax for
> lambda paramters"
> File > Settings > Build, Execution, Deployment > Build Tools > Maven >
> Importing
>   - JDK for importer = 1.8
>
> I go to File > Settings > Build, Execution, Deployment > Compiler > Java
> Compiler
>   - Project bytecode version = 8
>   - Select all modules and remove them
>   - Apply
> From Project view, select root level pom.xml > Maven > Reimport
> Return to File > Settings > Build, Execution, Deployment > Compiler > Java
> Compiler
>   - All modules have returned and have a Target bytecode version of 11
>
> At this point, I can't run unit tests in IntelliJ. I get an error "Error:
> java: invalid source release: 11"
>
> I could just use Java 11, but I'm curious if this is related to the jigsaw
> profile. When I remove the  from the jigsaw profile, I get the
> above behavior. When I remove both the activation and the properties
> (maven.compiler.sorce and maven.compiler.target), then I can get the
> language level to remain at 1.8.
>
> It appears the jigaw profile may be activating all the time. Or, maybe
> IntelliJ is presenting the incorrect JDK version?
>
> -Mark
>

Re: IntelliJ, language level and jigaw profile

[Jeff]

Mark,

Glad to see that it worked for you.  I have more information stashed away
somewhere about how and why IntelliJ IDEA implicitly enables profiles, but
having Java 11 on your system and IDEA knowing about it seems to trigger
the implicit activation of the "jigsaw" profile.  You won't have the same
behavior through Maven on the command line unless you're building with Java
11, since the profile is set to auto-activate under that circumstance.
That may be why IDEA is implicitly activating it, even though you've set
the project to be built with Java 8.

It's a bit of a PITA to go through a build and then realize that the
"jigsaw" profile was implicitly checked.  I think there are some issues
filed about that kind of profile activation over at JetBrains, but I don't
think they've gotten much attention.

On Thu, Jan 30, 2020 at 9:19 AM Mark Bean  wrote:

> Found it. Profiles are at the top of the list in the Maven tool window.
>
> Thanks for the tip. Unchecking the jigsaw profile did the trick.
>
> Thanks,
> Mark
>
>
> On Thu, Jan 30, 2020 at 8:51 AM Mark Bean  wrote:
>
> > Jeff,
> >
> > Can you please be a little more specific where to find the light/bold
> > checked profile option? The only way I've enabled profiles is by explicit
> > maven option "-P".
> >
> > Thanks,
> > Mark
> >
> > On Wed, Jan 29, 2020 at 4:31 PM Jeff  wrote:
> >
> >> Hi Mark,
> >>
> >> IntelliJ IDEA seems to implicitly activate the "jigsaw" profile with a
> >> "light" checkmark if it thinks you have Java 11 installed versus a
> "bold"
> >> checkmark if you've clicked that profile to be enabled explicitly.  Make
> >> sure that profile is unchecked, and make IDEA rebuild.  That should get
> >> you
> >> back to having your classes compiled with Java 8.
> >>
> >> On Wed, Jan 29, 2020 at 4:23 PM Mark Bean 
> wrote:
> >>
> >> > I'm having trouble with IntelliJ setting the proper language level.
> Has
> >> > anyone else seen the following behavior?
> >> >
> >> > IntelliJ 2019.3.2 (Community Edition)
> >> > installed from ideaIC-2019.3.2-no-jbr.tar.gz
> >> >
> >> > File > Project Structure > Platform Settings > SDKs
> >> >   - Only 1.8 is loaded
> >> > File > Project Structure > Project Settings > Project
> >> >   - Project SDK = 1.8
> >> >   - Project language level = 8 - Lambdas, type annotations etc.
> >> > File > Project Structure > Project Settings > Modules
> >> >   - Every module lists "Language level" as "11 - Local variable syntax
> >> for
> >> > lambda paramters"
> >> > File > Settings > Build, Execution, Deployment > Build Tools > Maven >
> >> > Importing
> >> >   - JDK for importer = 1.8
> >> >
> >> > I go to File > Settings > Build, Execution, Deployment > Compiler >
> Java
> >> > Compiler
> >> >   - Project bytecode version = 8
> >> >   - Select all modules and remove them
> >> >   - Apply
> >> > From Project view, select root level pom.xml > Maven > Reimport
> >> > Return to File > Settings > Build, Execution, Deployment > Compiler >
> >> Java
> >> > Compiler
> >> >   - All modules have returned and have a Target bytecode version of 11
> >> >
> >> > At this point, I can't run unit tests in IntelliJ. I get an error
> >> "Error:
> >> > java: invalid source release: 11"
> >> >
> >> > I could just use Java 11, but I'm curious if this is related to the
> >> jigsaw
> >> > profile. When I remove the  from the jigsaw profile, I get
> >> the
> >> > above behavior. When I remove both the activation and the properties
> >> > (maven.compiler.sorce and maven.compiler.target), then I can get the
> >> > language level to remain at 1.8.
> >> >
> >> > It appears the jigaw profile may be activating all the time. Or, maybe
> >> > IntelliJ is presenting the incorrect JDK version?
> >> >
> >> > -Mark
> >> >
> >>
> >
>

Developer questions

[Jeff]

Hi, 
I trying to understand how Nifi is to be used in a large production 
environment.  Nifi has the concept of a graph where multiple processors / 
processor groups can be placed.  Is the idea to have all running processor on a 
single graph organized by processor groups?  Comparing to Informatica’s data 
integration tool, workflows are used to organize sessions and a session is an 
instance of a mapping and a mapping has one or more sources, targets and 
transformations.  From a user interface perspective, the Nifi graph is similar 
to Informatica’s workflow.  Informatica also has the concept of a project 
folder which is used to organize workflows, mappings, etc…   

Also, what is the process for making updates to a template?  

Thanks for your help.

-cb

nifi fails build on CentOS 7

[Jeff]

I’m getting the below on a VM. It does build successfully on my Mac, though.

mvn -e -X -T 2.0C clean install
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) on 
project nifi-api: Compilation failure -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) on 
project nifi-api: Compilation failure
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at 
org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:189)
at 
org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:185)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.maven.plugin.compiler.CompilationFailureException: 
Compilation failure
at 
org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:913)
at 
org.apache.maven.plugin.compiler.CompilerMojo.execute(CompilerMojo.java:129)
at 
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
... 11 more

Thanks,
Jeff

Re: nifi fails build on CentOS 7

[Jeff]

JAVA_HOME is currently not set but I’ve tried both ways.  

[root@localhost nifi]# java -version
openjdk version "1.8.0_65"
OpenJDK Runtime Environment (build 1.8.0_65-b17)
OpenJDK 64-Bit Server VM (build 25.65-b01, mixed mode)


[root@localhost nifi]# mvn -version
which: no javac in 
(/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/opt/apache-maven-3.3.3/bin)
Warning: JAVA_HOME environment variable is not set.
Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06; 
2015-04-22T06:57:37-05:00)
Maven home: /opt/apache-maven-3.3.3
Java version: 1.8.0_65, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.65-2.b17.el7_1.x86_64/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-229.11.1.el7.x86_64", arch: "amd64", family: 
"unix"



> On Nov 3, 2015, at 5:37 PM, Joe Witt  wrote:
> 
> Jeff
> 
> When you run 'java -version' and 'mvn -version' what do you get back?
> 
> Thanks
> Joe
> 
> On Tue, Nov 3, 2015 at 11:35 PM, Jeff  wrote:
>> I’m getting the below on a VM. It does build successfully on my Mac, though.
>> 
>> mvn -e -X -T 2.0C clean install
>> [ERROR] Failed to execute goal 
>> org.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) 
>> on project nifi-api: Compilation failure -> [Help 1]
>> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
>> goal org.apache.maven.plugins:maven-compiler-plugin:3.2:compile 
>> (default-compile) on project nifi-api: Compilation failure
>>at 
>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
>>at 
>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>>at 
>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>>at 
>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>>at 
>> org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:189)
>>at 
>> org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:185)
>>at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>at 
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>at java.lang.Thread.run(Thread.java:745)
>> Caused by: org.apache.maven.plugin.compiler.CompilationFailureException: 
>> Compilation failure
>>at 
>> org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:913)
>>at 
>> org.apache.maven.plugin.compiler.CompilerMojo.execute(CompilerMojo.java:129)
>>at 
>> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
>>at 
>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
>>... 11 more
>> 
>> Thanks,
>> Jeff

Re: nifi fails build on CentOS 7

[Jeff]

The only other message I see is

[WARNING] Unable to autodetect 'javac' path, using 'javac' from the environment.

No worries, it can wait.

Thanks,

> On Nov 3, 2015, at 5:46 PM, Joe Witt  wrote:
> 
> if i weren't on horrible hotel wifi right now i'd try to pull down
> centos.  Is there anymore that can be found with those stack traces .
> They're not saying much.  Your java and maven versions seem quite good
> so need to dig further.
> 
> On Tue, Nov 3, 2015 at 11:42 PM, Jeff  wrote:
>> JAVA_HOME is currently not set but I’ve tried both ways.
>> 
>> [root@localhost nifi]# java -version
>> openjdk version "1.8.0_65"
>> OpenJDK Runtime Environment (build 1.8.0_65-b17)
>> OpenJDK 64-Bit Server VM (build 25.65-b01, mixed mode)
>> 
>> 
>> [root@localhost nifi]# mvn -version
>> which: no javac in 
>> (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/opt/apache-maven-3.3.3/bin)
>> Warning: JAVA_HOME environment variable is not set.
>> Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06; 
>> 2015-04-22T06:57:37-05:00)
>> Maven home: /opt/apache-maven-3.3.3
>> Java version: 1.8.0_65, vendor: Oracle Corporation
>> Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.65-2.b17.el7_1.x86_64/jre
>> Default locale: en_US, platform encoding: UTF-8
>> OS name: "linux", version: "3.10.0-229.11.1.el7.x86_64", arch: "amd64", 
>> family: "unix"
>> 
>> 
>> 
>>> On Nov 3, 2015, at 5:37 PM, Joe Witt  wrote:
>>> 
>>> Jeff
>>> 
>>> When you run 'java -version' and 'mvn -version' what do you get back?
>>> 
>>> Thanks
>>> Joe
>>> 
>>> On Tue, Nov 3, 2015 at 11:35 PM, Jeff  wrote:
>>>> I’m getting the below on a VM. It does build successfully on my Mac, 
>>>> though.
>>>> 
>>>> mvn -e -X -T 2.0C clean install
>>>> [ERROR] Failed to execute goal 
>>>> org.apache.maven.plugins:maven-compiler-plugin:3.2:compile 
>>>> (default-compile) on project nifi-api: Compilation failure -> [Help 1]
>>>> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
>>>> goal org.apache.maven.plugins:maven-compiler-plugin:3.2:compile 
>>>> (default-compile) on project nifi-api: Compilation failure
>>>>   at 
>>>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
>>>>   at 
>>>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>>>>   at 
>>>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>>>>   at 
>>>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>>>>   at 
>>>> org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:189)
>>>>   at 
>>>> org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:185)
>>>>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>   at 
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>   at 
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>   at 
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>   at java.lang.Thread.run(Thread.java:745)
>>>> Caused by: org.apache.maven.plugin.compiler.CompilationFailureException: 
>>>> Compilation failure
>>>>   at 
>>>> org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:913)
>>>>   at 
>>>> org.apache.maven.plugin.compiler.CompilerMojo.execute(CompilerMojo.java:129)
>>>>   at 
>>>> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
>>>>   at 
>>>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
>>>>   ... 11 more
>>>> 
>>>> Thanks,
>>>> Jeff
>> 

Re: nifi fails build on CentOS 7

[Jeff]

[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 8.091 s
[INFO] Finished at: 2015-11-03T12:08:59-06:00
[INFO] Final Memory: 76M/365M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) on 
project nifi-api: Compilation failure -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :nifi-api



> On Nov 3, 2015, at 6:07 PM, Joe Witt  wrote:
> 
> Can you go ahead and try just plain ol'
> 
> mvn clean install
> 
> I am curious if there is a memory/settings issue that is manifesting
> as an exception during compilation but that it isn't actually a
> compilation problem at all...
> 
> On Wed, Nov 4, 2015 at 12:06 AM, Jeff  wrote:
>> The only other message I see is
>> 
>> [WARNING] Unable to autodetect 'javac' path, using 'javac' from the 
>> environment.
>> 
>> No worries, it can wait.
>> 
>> Thanks,
>> 
>>> On Nov 3, 2015, at 5:46 PM, Joe Witt  wrote:
>>> 
>>> if i weren't on horrible hotel wifi right now i'd try to pull down
>>> centos.  Is there anymore that can be found with those stack traces .
>>> They're not saying much.  Your java and maven versions seem quite good
>>> so need to dig further.
>>> 
>>> On Tue, Nov 3, 2015 at 11:42 PM, Jeff  wrote:
>>>> JAVA_HOME is currently not set but I’ve tried both ways.
>>>> 
>>>> [root@localhost nifi]# java -version
>>>> openjdk version "1.8.0_65"
>>>> OpenJDK Runtime Environment (build 1.8.0_65-b17)
>>>> OpenJDK 64-Bit Server VM (build 25.65-b01, mixed mode)
>>>> 
>>>> 
>>>> [root@localhost nifi]# mvn -version
>>>> which: no javac in 
>>>> (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/opt/apache-maven-3.3.3/bin)
>>>> Warning: JAVA_HOME environment variable is not set.
>>>> Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06; 
>>>> 2015-04-22T06:57:37-05:00)
>>>> Maven home: /opt/apache-maven-3.3.3
>>>> Java version: 1.8.0_65, vendor: Oracle Corporation
>>>> Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.65-2.b17.el7_1.x86_64/jre
>>>> Default locale: en_US, platform encoding: UTF-8
>>>> OS name: "linux", version: "3.10.0-229.11.1.el7.x86_64", arch: "amd64", 
>>>> family: "unix"
>>>> 
>>>> 
>>>> 
>>>>> On Nov 3, 2015, at 5:37 PM, Joe Witt  wrote:
>>>>> 
>>>>> Jeff
>>>>> 
>>>>> When you run 'java -version' and 'mvn -version' what do you get back?
>>>>> 
>>>>> Thanks
>>>>> Joe
>>>>> 
>>>>> On Tue, Nov 3, 2015 at 11:35 PM, Jeff  wrote:
>>>>>> I’m getting the below on a VM. It does build successfully on my Mac, 
>>>>>> though.
>>>>>> 
>>>>>> mvn -e -X -T 2.0C clean install
>>>>>> [ERROR] Failed to execute goal 
>>>>>> org.apache.maven.plugins:maven-compiler-plugin:3.2:compile 
>>>>>> (default-compile) on project nifi-api: Compilation failure -> [Help 1]
>>>>>> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to 
>>>>>> execute goal org.apache.maven.plugins:maven-compiler-plugin:3.2:compile 
>>>>>> (default-compile) on project nifi-api: Compilation failure
>>>>>>  at 
>>>>>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
>>>>>>  at 
>>>>>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>>>>>>  at 
>>>>>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>>>>>>  at 
>>>>>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuil

Re: nifi fails build on CentOS 7

[Jeff]

find / -name javac did not return anything

I had JAVA_HOME set to 
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.65-2.b17.el7_1.x86_64/jre



> On Nov 3, 2015, at 6:30 PM, Tony Kurc  wrote:
> 
> Jeff,
> Were you able to run that find command?
> 
> On Tue, Nov 3, 2015 at 7:09 PM, Jeff  wrote:
> 
>> 
>> [INFO] BUILD FAILURE
>> [INFO]
>> 
>> [INFO] Total time: 8.091 s
>> [INFO] Finished at: 2015-11-03T12:08:59-06:00
>> [INFO] Final Memory: 76M/365M
>> [INFO]
>> 
>> [ERROR] Failed to execute goal
>> org.apache.maven.plugins:maven-compiler-plugin:3.2:compile
>> (default-compile) on project nifi-api: Compilation failure -> [Help 1]
>> [ERROR]
>> [ERROR] To see the full stack trace of the errors, re-run Maven with the
>> -e switch.
>> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
>> [ERROR]
>> [ERROR] For more information about the errors and possible solutions,
>> please read the following articles:
>> [ERROR] [Help 1]
>> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
>> [ERROR]
>> [ERROR] After correcting the problems, you can resume the build with the
>> command
>> [ERROR]   mvn  -rf :nifi-api
>> 
>> 
>> 
>>> On Nov 3, 2015, at 6:07 PM, Joe Witt  wrote:
>>> 
>>> Can you go ahead and try just plain ol'
>>> 
>>> mvn clean install
>>> 
>>> I am curious if there is a memory/settings issue that is manifesting
>>> as an exception during compilation but that it isn't actually a
>>> compilation problem at all...
>>> 
>>> On Wed, Nov 4, 2015 at 12:06 AM, Jeff  wrote:
>>>> The only other message I see is
>>>> 
>>>> [WARNING] Unable to autodetect 'javac' path, using 'javac' from the
>> environment.
>>>> 
>>>> No worries, it can wait.
>>>> 
>>>> Thanks,
>>>> 
>>>>> On Nov 3, 2015, at 5:46 PM, Joe Witt  wrote:
>>>>> 
>>>>> if i weren't on horrible hotel wifi right now i'd try to pull down
>>>>> centos.  Is there anymore that can be found with those stack traces .
>>>>> They're not saying much.  Your java and maven versions seem quite good
>>>>> so need to dig further.
>>>>> 
>>>>> On Tue, Nov 3, 2015 at 11:42 PM, Jeff  wrote:
>>>>>> JAVA_HOME is currently not set but I’ve tried both ways.
>>>>>> 
>>>>>> [root@localhost nifi]# java -version
>>>>>> openjdk version "1.8.0_65"
>>>>>> OpenJDK Runtime Environment (build 1.8.0_65-b17)
>>>>>> OpenJDK 64-Bit Server VM (build 25.65-b01, mixed mode)
>>>>>> 
>>>>>> 
>>>>>> [root@localhost nifi]# mvn -version
>>>>>> which: no javac in
>> (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/opt/apache-maven-3.3.3/bin)
>>>>>> Warning: JAVA_HOME environment variable is not set.
>>>>>> Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06;
>> 2015-04-22T06:57:37-05:00)
>>>>>> Maven home: /opt/apache-maven-3.3.3
>>>>>> Java version: 1.8.0_65, vendor: Oracle Corporation
>>>>>> Java home:
>> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.65-2.b17.el7_1.x86_64/jre
>>>>>> Default locale: en_US, platform encoding: UTF-8
>>>>>> OS name: "linux", version: "3.10.0-229.11.1.el7.x86_64", arch:
>> "amd64", family: "unix"
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> On Nov 3, 2015, at 5:37 PM, Joe Witt  wrote:
>>>>>>> 
>>>>>>> Jeff
>>>>>>> 
>>>>>>> When you run 'java -version' and 'mvn -version' what do you get back?
>>>>>>> 
>>>>>>> Thanks
>>>>>>> Joe
>>>>>>> 
>>>>>>> On Tue, Nov 3, 2015 at 11:35 PM, Jeff  wrote:
>>>>>>>> I’m getting the below on a VM. It does build successfully on my
>> Mac, though.
>>>>>>>> 
>>>>>>>> mvn -e -X -T 2.0C clean install
>>>>>>>> [ERROR] Failed to execute goal
>> org.apache.ma

Re: nifi fails build on CentOS 7

[Jeff]

I removed openjdk and installed the Oracle jdk and it now works.

Thanks for the help!!

> On Nov 3, 2015, at 6:40 PM, Tony Kurc  wrote:
> 
> it is entirely possible that you only have the jre installed. 'yum
> whatprovides "*/bin/javac"' may tell you packages which you can install
> which include a compiler.
> 
> On Tue, Nov 3, 2015 at 7:35 PM, Jeff  wrote:
> 
>> 
>> 
>> find / -name javac did not return anything
>> 
>> I had JAVA_HOME set to
>> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.65-2.b17.el7_1.x86_64/jre
>> 
>> 
>> 
>>> On Nov 3, 2015, at 6:30 PM, Tony Kurc  wrote:
>>> 
>>> Jeff,
>>> Were you able to run that find command?
>>> 
>>> On Tue, Nov 3, 2015 at 7:09 PM, Jeff  wrote:
>>> 
>>>> 
>>>> [INFO] BUILD FAILURE
>>>> [INFO]
>>>> 
>>>> [INFO] Total time: 8.091 s
>>>> [INFO] Finished at: 2015-11-03T12:08:59-06:00
>>>> [INFO] Final Memory: 76M/365M
>>>> [INFO]
>>>> 
>>>> [ERROR] Failed to execute goal
>>>> org.apache.maven.plugins:maven-compiler-plugin:3.2:compile
>>>> (default-compile) on project nifi-api: Compilation failure -> [Help 1]
>>>> [ERROR]
>>>> [ERROR] To see the full stack trace of the errors, re-run Maven with the
>>>> -e switch.
>>>> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
>>>> [ERROR]
>>>> [ERROR] For more information about the errors and possible solutions,
>>>> please read the following articles:
>>>> [ERROR] [Help 1]
>>>> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
>>>> [ERROR]
>>>> [ERROR] After correcting the problems, you can resume the build with the
>>>> command
>>>> [ERROR]   mvn  -rf :nifi-api
>>>> 
>>>> 
>>>> 
>>>>> On Nov 3, 2015, at 6:07 PM, Joe Witt  wrote:
>>>>> 
>>>>> Can you go ahead and try just plain ol'
>>>>> 
>>>>> mvn clean install
>>>>> 
>>>>> I am curious if there is a memory/settings issue that is manifesting
>>>>> as an exception during compilation but that it isn't actually a
>>>>> compilation problem at all...
>>>>> 
>>>>> On Wed, Nov 4, 2015 at 12:06 AM, Jeff  wrote:
>>>>>> The only other message I see is
>>>>>> 
>>>>>> [WARNING] Unable to autodetect 'javac' path, using 'javac' from the
>>>> environment.
>>>>>> 
>>>>>> No worries, it can wait.
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>>> On Nov 3, 2015, at 5:46 PM, Joe Witt  wrote:
>>>>>>> 
>>>>>>> if i weren't on horrible hotel wifi right now i'd try to pull down
>>>>>>> centos.  Is there anymore that can be found with those stack traces .
>>>>>>> They're not saying much.  Your java and maven versions seem quite
>> good
>>>>>>> so need to dig further.
>>>>>>> 
>>>>>>> On Tue, Nov 3, 2015 at 11:42 PM, Jeff  wrote:
>>>>>>>> JAVA_HOME is currently not set but I’ve tried both ways.
>>>>>>>> 
>>>>>>>> [root@localhost nifi]# java -version
>>>>>>>> openjdk version "1.8.0_65"
>>>>>>>> OpenJDK Runtime Environment (build 1.8.0_65-b17)
>>>>>>>> OpenJDK 64-Bit Server VM (build 25.65-b01, mixed mode)
>>>>>>>> 
>>>>>>>> 
>>>>>>>> [root@localhost nifi]# mvn -version
>>>>>>>> which: no javac in
>>>> 
>> (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/opt/apache-maven-3.3.3/bin)
>>>>>>>> Warning: JAVA_HOME environment variable is not set.
>>>>>>>> Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06;
>>>> 2015-04-22T06:57:37-05:00)
>>>>>>>> Maven home: /opt/apache-maven-3.3.3
>>>>>>>> Java version: 1.8.0_65, vendor: Oracle Corporation
>>>>>>>

Re: nifi fails build on CentOS 7

[Jeff]

Well, the build got much further with the Oracle JDK but still failed;

[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 07:47 min (Wall Clock)
[INFO] Finished at: 2015-11-03T17:07:40-06:00
[INFO] Final Memory: 183M/564M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.18:test (default-test) on 
project nifi-framework-core: There are test failures.
[ERROR] 
[ERROR] Please refer to 
/root/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/target/surefire-reports
 for the individual test results.



> On Nov 3, 2015, at 6:54 PM, Cerulean Blue  wrote:
> 
> Oh, I also adjusted the memory!
> 
> Sent from my iPhone
> 
>> On Nov 3, 2015, at 6:50 PM, Joe Witt  wrote:
>> 
>> Jeff - can you please send another note telling Tony that it turns out
>> it was memory settings :-)
>> 
>> He is making fun of me off-list.  He should totally do that on-list!
>> 
>> Glad you're in business now and sorry for my terrible misdirection.
>> 
>> Joe
>> 
>>> On Wed, Nov 4, 2015 at 12:49 AM, Tony Kurc  wrote:
>>> Huzzah
>>> 
>>>> On Tue, Nov 3, 2015 at 7:48 PM, Jeff  wrote:
>>>> 
>>>> 
>>>> I removed openjdk and installed the Oracle jdk and it now works.
>>>> 
>>>> Thanks for the help!!
>>>> 
>>>>> On Nov 3, 2015, at 6:40 PM, Tony Kurc  wrote:
>>>>> 
>>>>> it is entirely possible that you only have the jre installed. 'yum
>>>>> whatprovides "*/bin/javac"' may tell you packages which you can install
>>>>> which include a compiler.
>>>>> 
>>>>>> On Tue, Nov 3, 2015 at 7:35 PM, Jeff  wrote:
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> find / -name javac did not return anything
>>>>>> 
>>>>>> I had JAVA_HOME set to
>>>>>> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.65-2.b17.el7_1.x86_64/jre
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> On Nov 3, 2015, at 6:30 PM, Tony Kurc  wrote:
>>>>>>> 
>>>>>>> Jeff,
>>>>>>> Were you able to run that find command?
>>>>>>> 
>>>>>>>> On Tue, Nov 3, 2015 at 7:09 PM, Jeff  wrote:
>>>>>>>> 
>>>>>>>> 
>>>>>>>> [INFO] BUILD FAILURE
>>>>>>>> [INFO]
>>>> 
>>>>>>>> [INFO] Total time: 8.091 s
>>>>>>>> [INFO] Finished at: 2015-11-03T12:08:59-06:00
>>>>>>>> [INFO] Final Memory: 76M/365M
>>>>>>>> [INFO]
>>>> 
>>>>>>>> [ERROR] Failed to execute goal
>>>>>>>> org.apache.maven.plugins:maven-compiler-plugin:3.2:compile
>>>>>>>> (default-compile) on project nifi-api: Compilation failure -> [Help 1]
>>>>>>>> [ERROR]
>>>>>>>> [ERROR] To see the full stack trace of the errors, re-run Maven with
>>>> the
>>>>>>>> -e switch.
>>>>>>>> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
>>>>>>>> [ERROR]
>>>>>>>> [ERROR] For more information about the errors and possible solutions,
>>>>>>>> please read the following articles:
>>>>>>>> [ERROR] [Help 1]
>>>>>>>> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
>>>>>>>> [ERROR]
>>>>>>>> [ERROR] After correcting the problems, you can resume the build with
>>>> the
>>>>>>>> command
>>>>>>>> [ERROR]   mvn  -rf :nifi-api
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Nov 3, 2015, at 6:07 PM, Joe Witt  wrote:
>>>>>>>>> 
>>>>>>>>> Can you go ahead and try just plain ol'
>>>>>>>>> 
>>>>>>>>> m

Re: nifi fails build on CentOS 7

[Jeff]

Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 28.916 sec <<< 
FAILURE! - in org.apache.nifi.controller.TestStandardFlowFileQueue
testDropSwappedFlowFiles(org.apache.nifi.controller.TestStandardFlowFileQueue)  
Time elapsed: 21.825 sec  <<< ERROR!
org.junit.runners.model.TestTimedOutException: test timed out after 2 
milliseconds
at java.lang.Throwable.fillInStackTrace(Native Method)
at java.lang.Throwable.fillInStackTrace(Throwable.java:783)
at java.lang.Throwable.(Throwable.java:250)
at 
org.mockito.internal.debugging.LocationImpl.(LocationImpl.java:24)
at 
org.mockito.internal.debugging.LocationImpl.(LocationImpl.java:19)
at 
org.mockito.internal.invocation.InvocationImpl.(InvocationImpl.java:50)
at 
org.mockito.internal.creation.cglib.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:58)
at 
org.apache.nifi.connectable.Connection$$EnhancerByMockitoWithCGLIB$$8aa704ad.getDestination()
at 
org.apache.nifi.controller.StandardFlowFileQueue.put(StandardFlowFileQueue.java:315)
at 
org.apache.nifi.controller.TestStandardFlowFileQueue.testDropSwappedFlowFiles(TestStandardFlowFileQueue.java:169)

> On Nov 4, 2015, at 7:51 AM, Brandon DeVries  wrote:
> 
> Jeff,
> 
> Can you see what tests had failures?  Also, if you just want to verify that
> your system can do the build, you could do something like "mvn clean
> install -DskipTests".  Let us know what you see.  Thanks.
> 
> Brandon
> 
> On Wed, Nov 4, 2015 at 8:46 AM, Jeff  wrote:
> 
>> 
>> Well, the build got much further with the Oracle JDK but still failed;
>> 
>> [INFO]
>> 
>> [INFO] BUILD FAILURE
>> [INFO]
>> 
>> [INFO] Total time: 07:47 min (Wall Clock)
>> [INFO] Finished at: 2015-11-03T17:07:40-06:00
>> [INFO] Final Memory: 183M/564M
>> [INFO]
>> 
>> [ERROR] Failed to execute goal
>> org.apache.maven.plugins:maven-surefire-plugin:2.18:test (default-test) on
>> project nifi-framework-core: There are test failures.
>> [ERROR]
>> [ERROR] Please refer to
>> /root/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/target/surefire-reports
>> for the individual test results.
>> 
>> 
>> 
>>> On Nov 3, 2015, at 6:54 PM, Cerulean Blue  wrote:
>>> 
>>> Oh, I also adjusted the memory!
>>> 
>>> Sent from my iPhone
>>> 
>>>> On Nov 3, 2015, at 6:50 PM, Joe Witt  wrote:
>>>> 
>>>> Jeff - can you please send another note telling Tony that it turns out
>>>> it was memory settings :-)
>>>> 
>>>> He is making fun of me off-list.  He should totally do that on-list!
>>>> 
>>>> Glad you're in business now and sorry for my terrible misdirection.
>>>> 
>>>> Joe
>>>> 
>>>>> On Wed, Nov 4, 2015 at 12:49 AM, Tony Kurc  wrote:
>>>>> Huzzah
>>>>> 
>>>>>> On Tue, Nov 3, 2015 at 7:48 PM, Jeff  wrote:
>>>>>> 
>>>>>> 
>>>>>> I removed openjdk and installed the Oracle jdk and it now works.
>>>>>> 
>>>>>> Thanks for the help!!
>>>>>> 
>>>>>>> On Nov 3, 2015, at 6:40 PM, Tony Kurc  wrote:
>>>>>>> 
>>>>>>> it is entirely possible that you only have the jre installed. 'yum
>>>>>>> whatprovides "*/bin/javac"' may tell you packages which you can
>> install
>>>>>>> which include a compiler.
>>>>>>> 
>>>>>>>> On Tue, Nov 3, 2015 at 7:35 PM, Jeff  wrote:
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> find / -name javac did not return anything
>>>>>>>> 
>>>>>>>> I had JAVA_HOME set to
>>>>>>>> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.65-2.b17.el7_1.x86_64/jre
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Nov 3, 2015, at 6:30 PM, Tony Kurc  wrote:
>>>>>>>>> 
>>>>>>>>> Jeff,
>>>>>>>>> Were you able to run that find command?
>>>>>>>>> 
>>>>>>>>&g

Re: nifi fails build on CentOS 7

[Jeff]

Yes, the build succeeded using;

mvn clean install -DskipTests

[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 07:27 min
[INFO] Finished at: 2015-11-04T02:02:57-06:00
[INFO] Final Memory: 197M/566M
[INFO] 

> On Nov 4, 2015, at 7:51 AM, Brandon DeVries  wrote:
> 
> mvn clean
> install -DskipTests

Re: nifi fails build on CentOS 7

[Jeff]

the only items under

.plxarc
classes
generated-sources
generated-test-sources
maven-archiver
maven-shared-archive-resources
maven-status
nifi-framework-core-0.3.1-SNAPSHOT.jar
test-classes

I searched for *FileQueue*.txt but no files found.

I can create a new VM and try again.  I do have a few things on this server.


> On Nov 4, 2015, at 9:01 AM, Mark Payne  wrote:
> 
> Jeff,
> 
> I just checked out a fresh build from master, running on CentOS 7 and was 
> able to build without any problems.
> So I don't think it is specifically a problem with CentOS 7. Can you provide 
> the output of the unit tests?
> 
> They should be in 
> ${NIFI_SRC_DIR}/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/target/surefire-reports
> 
> The two files of interest are 
> org.apache.nifi.controller.TestStandardFlowFileQueue.txt and 
> org.apache.nifi.controller.TestStandardFlowFileQueue-output.txt
> 
> Thanks!
> -Mark
> 
> 
>> On Nov 4, 2015, at 8:53 AM, Jeff  wrote:
>> 
>> Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 28.916 sec 
>> <<< FAILURE! - in org.apache.nifi.controller.TestStandardFlowFileQueue
>> testDropSwappedFlowFiles(org.apache.nifi.controller.TestStandardFlowFileQueue)
>>   Time elapsed: 21.825 sec  <<< ERROR!
>> org.junit.runners.model.TestTimedOutException: test timed out after 2 
>> milliseconds
>>  at java.lang.Throwable.fillInStackTrace(Native Method)
>>  at java.lang.Throwable.fillInStackTrace(Throwable.java:783)
>>  at java.lang.Throwable.(Throwable.java:250)
>>  at 
>> org.mockito.internal.debugging.LocationImpl.(LocationImpl.java:24)
>>  at 
>> org.mockito.internal.debugging.LocationImpl.(LocationImpl.java:19)
>>  at 
>> org.mockito.internal.invocation.InvocationImpl.(InvocationImpl.java:50)
>>  at 
>> org.mockito.internal.creation.cglib.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:58)
>>  at 
>> org.apache.nifi.connectable.Connection$$EnhancerByMockitoWithCGLIB$$8aa704ad.getDestination()
>>  at 
>> org.apache.nifi.controller.StandardFlowFileQueue.put(StandardFlowFileQueue.java:315)
>>  at 
>> org.apache.nifi.controller.TestStandardFlowFileQueue.testDropSwappedFlowFiles(TestStandardFlowFileQueue.java:169)
>> 
>>> On Nov 4, 2015, at 7:51 AM, Brandon DeVries  wrote:
>>> 
>>> Jeff,
>>> 
>>> Can you see what tests had failures?  Also, if you just want to verify that
>>> your system can do the build, you could do something like "mvn clean
>>> install -DskipTests".  Let us know what you see.  Thanks.
>>> 
>>> Brandon
>>> 
>>> On Wed, Nov 4, 2015 at 8:46 AM, Jeff  wrote:
>>> 
>>>> 
>>>> Well, the build got much further with the Oracle JDK but still failed;
>>>> 
>>>> [INFO]
>>>> 
>>>> [INFO] BUILD FAILURE
>>>> [INFO]
>>>> 
>>>> [INFO] Total time: 07:47 min (Wall Clock)
>>>> [INFO] Finished at: 2015-11-03T17:07:40-06:00
>>>> [INFO] Final Memory: 183M/564M
>>>> [INFO]
>>>> 
>>>> [ERROR] Failed to execute goal
>>>> org.apache.maven.plugins:maven-surefire-plugin:2.18:test (default-test) on
>>>> project nifi-framework-core: There are test failures.
>>>> [ERROR]
>>>> [ERROR] Please refer to
>>>> /root/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/target/surefire-reports
>>>> for the individual test results.
>>>> 
>>>> 
>>>> 
>>>>> On Nov 3, 2015, at 6:54 PM, Cerulean Blue  wrote:
>>>>> 
>>>>> Oh, I also adjusted the memory!
>>>>> 
>>>>> Sent from my iPhone
>>>>> 
>>>>>> On Nov 3, 2015, at 6:50 PM, Joe Witt  wrote:
>>>>>> 
>>>>>> Jeff - can you please send another note telling Tony that it turns out
>>>>>> it was memory settings :-)
>>>>>> 
>>>>>> He is making fun of me off-list.  He should totally do that on-list!
>>>>>> 
>>>>>> Glad you're in business now and sorry for my terrible misdirection.
>>>>>> 
>>>>>> Joe
>>>>>> 
>>>>>>

Graph issue when adding a processor

[Jeff]

I just started building a flow that will read tab delimited files and converts 
to Avro.  I’m working with a 0.3.1 snapshot.  

I added few processors, then deleted, made a few connection changes.  When I 
went to add another processor, the processor did not add as expected.  I simply 
have the below icon on the graph.

I can not delete it or add any other processor.  

This icon is on all levels of the graph



Thanks,

Jeff

Re: Graph issue when adding a processor

[Jeff]

I’m running Centos7 and Chrome 45.0.2454.85 


> On Nov 5, 2015, at 9:21 AM, Matt Gilman  wrote:
> 
> Another NiFi user reported something very similar [1]. He was running Centos7 
> and using FireFox 31. He upgraded to FireFox 38 (which was available through 
> yum) and the problem went away.
> 
> Are you running in a similar environment?
> 
> Matt
> 
> [1] https://issues.apache.org/jira/browse/NIFI-956 
> <https://issues.apache.org/jira/browse/NIFI-956>
> 
> On Thu, Nov 5, 2015 at 10:17 AM, Jeff  <mailto:j.007...@gmail.com>> wrote:
> I just started building a flow that will read tab delimited files and 
> converts to Avro.  I’m working with a 0.3.1 snapshot.  
> 
> I added few processors, then deleted, made a few connection changes.  When I 
> went to add another processor, the processor did not add as expected.  I 
> simply have the below icon on the graph.
> 
> I can not delete it or add any other processor.  
> 
> This icon is on all levels of the graph
> 
> 
> 
> Thanks,
> 
> Jeff
> 

Re: Graph issue when adding a processor

[Jeff]

Using FireFox 38.2.1 seems to be fine


> On Nov 5, 2015, at 9:21 AM, Matt Gilman  wrote:
> 
> Another NiFi user reported something very similar [1]. He was running Centos7 
> and using FireFox 31. He upgraded to FireFox 38 (which was available through 
> yum) and the problem went away.
> 
> Are you running in a similar environment?
> 
> Matt
> 
> [1] https://issues.apache.org/jira/browse/NIFI-956 
> <https://issues.apache.org/jira/browse/NIFI-956>
> 
> On Thu, Nov 5, 2015 at 10:17 AM, Jeff  <mailto:j.007...@gmail.com>> wrote:
> I just started building a flow that will read tab delimited files and 
> converts to Avro.  I’m working with a 0.3.1 snapshot.  
> 
> I added few processors, then deleted, made a few connection changes.  When I 
> went to add another processor, the processor did not add as expected.  I 
> simply have the below icon on the graph.
> 
> I can not delete it or add any other processor.  
> 
> This icon is on all levels of the graph
> 
> 
> 
> Thanks,
> 
> Jeff
> 

JSON / Avro issues

[Jeff]

I built a simple flow that reads a tab separated file and attempts to convert 
to Avro.

ConvertCSVtoAvro just says that the conversion failed.  

Where can I find more information on what the failure was? 

Using the same sample tab separated file, I create a JSON file out of it.  

The JSON to Avro processor also fails with very little explication.


With regard to the ConvertCSVtoAvro processor
Since my file is tab  delimited, do I simple open the "CSV delimiter” 
property, delete , and hit the tab key or is there a special syntax like ^t?
My data has no CSV quote character so do I leave this as “or delete it 
or check the empty box?  

With regard to the ConvertJSONtoAvro
What is the expected JSON source file to look like?
[
 {fields values … },
 {fields values …}
]
Or
 {fields values … }
 {fields values …}
or something else.

Thanks,

Sorry for send this to 2 lists

Re: nifi fails build on CentOS 7

[Jeff]

I built a new VM CentOS7 

I get the below most of the time though last night it did succeeded.  

[INFO] nifi-framework-core  FAILURE [02:41 min]

[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 08:51 min (Wall Clock)
[INFO] Finished at: 2015-11-05T16:09:51-06:00
[INFO] Final Memory: 183M/514M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.18:test (default-test) on 
project nifi-framework-core: There are test failures.
[ERROR] 
[ERROR] Please refer to 
/root/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/target/surefire-reports
 for the individual test results.
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.18:test (default-test) on 
project nifi-framework-core: There are test failures.

Please refer to 
/root/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/target/surefire-reports
 for the individual test results.
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at 
org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:189)
at 
org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:185)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.maven.plugin.MojoFailureException: There are test 
failures.

Please refer to 
/root/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/target/surefire-reports
 for the individual test results.
at 
org.apache.maven.plugin.surefire.SurefireHelper.reportExecution(SurefireHelper.java:82)
at 
org.apache.maven.plugin.surefire.SurefirePlugin.handleSummary(SurefirePlugin.java:213)
at 
org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:883)
at 
org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:751)
at 
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
... 11 more
[ERROR] 




> On Nov 4, 2015, at 9:01 AM, Mark Payne  wrote:
> 
> Jeff,
> 
> I just checked out a fresh build from master, running on CentOS 7 and was 
> able to build without any problems.
> So I don't think it is specifically a problem with CentOS 7. Can you provide 
> the output of the unit tests?
> 
> They should be in 
> ${NIFI_SRC_DIR}/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/target/surefire-reports
> 
> The two files of interest are 
> org.apache.nifi.controller.TestStandardFlowFileQueue.txt and 
> org.apache.nifi.controller.TestStandardFlowFileQueue-output.txt
> 
> Thanks!
> -Mark
> 
> 
>> On Nov 4, 2015, at 8:53 AM, Jeff  wrote:
>> 
>> Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 28.916 sec 
>> <<< FAILURE! - in org.apache.nifi.controller.TestStandardFlowFileQueue
>> testDropSwappedFlowFiles(org.apache.nifi.controller.TestStandardFlowFileQueue)
>>   Time elapsed: 21.825 sec  <<< ERROR!
>> org.junit.runners.model.TestTimedOutException: test timed out after 2 
>> milliseconds
>>  at java.lang.Throwable.fillInStackTrace(Native Method)
>>  at java.lang.Throwable.fillInStackTrace(Throwable.java:783)
>>  at java.lang.Throwable.(Throwable.java:250)
>>  at 
>> org.mockito.internal.debugging.LocationImpl.(LocationImpl.java:24)
>>  at 
>> org.mockito.internal.debugging.LocationImpl.(LocationImpl.java:19)
>>  at 
>> org.mockito.internal.invocation.InvocationImpl.(InvocationImpl.java:50)
>>  at 
>> org.mockito.internal.c

Re: nifi fails build on CentOS 7

[Jeff]

I just cloned Nifi but I’m still seeing these errors when I do mvn -e -X -T 
2.0C clean install

[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.18:test (default-test) on 
project nifi-hdfs-processors: There are test failures.
[ERROR] 
[ERROR] Please refer to 
/root/nifi/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/target/surefire-reports
 for the individual test results.
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.18:test (default-test) on 
project nifi-hdfs-processors: There are test failures.

Please refer to 
/root/nifi/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/target/surefire-reports
 for the individual test results.
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at 
org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:189)
at 
org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:185)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.maven.plugin.MojoFailureException: There are test 
failures.

Please refer to 
/root/nifi/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/target/surefire-reports
 for the individual test results.
at 
org.apache.maven.plugin.surefire.SurefireHelper.reportExecution(SurefireHelper.java:82)
at 
org.apache.maven.plugin.surefire.SurefirePlugin.handleSummary(SurefirePlugin.java:213)
at 
org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:883)
at 
org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:751)
at 
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
... 11 more
[ERROR] 
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :nifi-hdfs-processors



> On Nov 10, 2015, at 12:33 PM, Aldrin Piri  wrote:
> 
> Seems there is in fact an issue with the test cited prior.  I received the
> following on my Travis CI build on a branch which has the latest of master
> merged into it.   Unfortunately, it seems that the environment is not the
> same as before.
> 
> In this case, Travis CI is running with Ubuntu 12.04 LTS [1].  For the
> particular build [2], Oracle JDK 7 built without issue, but Oracle JDK 8
> exhibited the same problem [3] as reported by Jeff.
> 
> NIFI-1145 was created to track this issue [4]
> 
> [1]
> http://docs.travis-ci.com/user/ci-environment/#Virtualization-environments
> [2] https://travis-ci.org/apiri/incubator-nifi/builds/90353191
> [3] https://s3.amazonaws.com/archive.travis-ci.org/jobs/90353192/log.txt
> [4] https://issues.apache.org/jira/browse/NIFI-1145
> 
> On Thu, Nov 5, 2015 at 5:17 PM, Jeff  wrote:
> 
>> 
>> 
>> I built a new VM CentOS7
>> 
>> I get the below most of the time though last night it did succeeded.
>> 
>> [INFO] nifi-framework-core  FAILURE [02:41
>> min]
>> 
>> [INFO] BUILD FAILURE
>> [INFO]
>> 
>> [INFO] Total time: 08:51 min (Wall Clock)
>> [INFO] Finished at: 2015-11-05T16:09:51-06:00
>> [INFO] Final Memory: 183M/514M
>> [INFO]
>> 
>> [ERROR] Failed to execute goal
>> org.apache.maven.plugins:maven-surefire-plugin:2.18:test (default-test) on
>> project nifi-framework-core: There are test failures.
>> [ERROR]
>> [ERROR] Please refer to
>> /root/nifi/nifi-nar-bundles/nifi

Re: nifi fails build on CentOS 7

[Jeff]

Yes, CentOS 7

Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.703 sec - in 
org.apache.nifi.processors.hadoop.TestListHDFS

Results :

Failed tests: 
  PutHDFSTest.testPutFileWithException:230 null



Tests run: 21, Failures: 1, Errors: 0, Skipped: 0



> On Dec 7, 2015, at 3:28 PM, Matt Gilman  wrote:
> 
> Jeff,
> 
> Are you still running on CentOS 7? Can you provide more details about the
> unit test is failing? Thanks!
> 
> Matt
> 
> On Mon, Dec 7, 2015 at 4:24 PM, Jeff  wrote:
> 
>> I just cloned Nifi but I’m still seeing these errors when I do mvn -e -X
>> -T 2.0C clean install
>> 
>> [ERROR] Failed to execute goal
>> org.apache.maven.plugins:maven-surefire-plugin:2.18:test (default-test) on
>> project nifi-hdfs-processors: There are test failures.
>> [ERROR]
>> [ERROR] Please refer to
>> /root/nifi/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/target/surefire-reports
>> for the individual test results.
>> [ERROR] -> [Help 1]
>> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
>> goal org.apache.maven.plugins:maven-surefire-plugin:2.18:test
>> (default-test) on project nifi-hdfs-processors: There are test failures.
>> 
>> Please refer to
>> /root/nifi/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/target/surefire-reports
>> for the individual test results.
>>at
>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
>>at
>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>>at
>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>>at
>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>>at
>> org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:189)
>>at
>> org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:185)
>>at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>at java.lang.Thread.run(Thread.java:745)
>> Caused by: org.apache.maven.plugin.MojoFailureException: There are test
>> failures.
>> 
>> Please refer to
>> /root/nifi/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/target/surefire-reports
>> for the individual test results.
>>at
>> org.apache.maven.plugin.surefire.SurefireHelper.reportExecution(SurefireHelper.java:82)
>>at
>> org.apache.maven.plugin.surefire.SurefirePlugin.handleSummary(SurefirePlugin.java:213)
>>at
>> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:883)
>>at
>> org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:751)
>>at
>> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
>>at
>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
>>... 11 more
>> [ERROR]
>> [ERROR]
>> [ERROR] For more information about the errors and possible solutions,
>> please read the following articles:
>> [ERROR] [Help 1]
>> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
>> [ERROR]
>> [ERROR] After correcting the problems, you can resume the build with the
>> command
>> [ERROR]   mvn  -rf :nifi-hdfs-processors
>> 
>> 
>> 
>>> On Nov 10, 2015, at 12:33 PM, Aldrin Piri  wrote:
>>> 
>>> Seems there is in fact an issue with the test cited prior.  I received
>> the
>>> following on my Travis CI build on a branch which has the latest of
>> master
>>> merged into it.   Unfortunately, it seems that the environment is not the
>>> same as before.
>>> 
>>> In this case, Travis CI is running with Ubuntu 12.04 LTS [1].  For the
>>> particular build [2], Oracle JDK 7 built without issue, but Oracle JDK 8
>>> exhibited the same problem [3] as reported by Jeff.
>>> 
>>> NIFI-1145 was created to track this issue [4]
>>> 
>>>

Re: nifi fails build on CentOS 7

[Jeff]

Hi Joe, 

You are welcome.  Below is what I’m seeing

---
Test set: org.apache.nifi.processors.hadoop.PutHDFSTest
---
Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.949 sec <<< 
FAILURE! - in org.apache.nifi.processors.hadoop.PutHDFSTest
testPutFileWithException(org.apache.nifi.processors.hadoop.PutHDFSTest)  Time 
elapsed: 0.532 sec  <<< FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertFalse(Assert.java:64)
at org.junit.Assert.assertFalse(Assert.java:74)
at 
org.apache.nifi.processors.hadoop.PutHDFSTest.testPutFileWithException(PutHDFSTest.java:230)

> On Dec 7, 2015, at 3:36 PM, Joe Witt  wrote:
> 
> thanks for reporting this Jeff and your timing could not have been better!
> 
> It's happening for Gilman as well.
> 
> Joe
> 
> On Mon, Dec 7, 2015 at 4:31 PM, Jeff  wrote:
>> Yes, CentOS 7
>> 
>> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.703 sec - 
>> in org.apache.nifi.processors.hadoop.TestListHDFS
>> 
>> Results :
>> 
>> Failed tests:
>>  PutHDFSTest.testPutFileWithException:230 null
>> 
>> 
>> 
>> Tests run: 21, Failures: 1, Errors: 0, Skipped: 0
>> 
>> 
>> 
>>> On Dec 7, 2015, at 3:28 PM, Matt Gilman  wrote:
>>> 
>>> Jeff,
>>> 
>>> Are you still running on CentOS 7? Can you provide more details about the
>>> unit test is failing? Thanks!
>>> 
>>> Matt
>>> 
>>> On Mon, Dec 7, 2015 at 4:24 PM, Jeff  wrote:
>>> 
>>>> I just cloned Nifi but I’m still seeing these errors when I do mvn -e -X
>>>> -T 2.0C clean install
>>>> 
>>>> [ERROR] Failed to execute goal
>>>> org.apache.maven.plugins:maven-surefire-plugin:2.18:test (default-test) on
>>>> project nifi-hdfs-processors: There are test failures.
>>>> [ERROR]
>>>> [ERROR] Please refer to
>>>> /root/nifi/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/target/surefire-reports
>>>> for the individual test results.
>>>> [ERROR] -> [Help 1]
>>>> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
>>>> goal org.apache.maven.plugins:maven-surefire-plugin:2.18:test
>>>> (default-test) on project nifi-hdfs-processors: There are test failures.
>>>> 
>>>> Please refer to
>>>> /root/nifi/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/target/surefire-reports
>>>> for the individual test results.
>>>>   at
>>>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
>>>>   at
>>>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>>>>   at
>>>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>>>>   at
>>>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>>>>   at
>>>> org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:189)
>>>>   at
>>>> org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:185)
>>>>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>   at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>   at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>   at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>   at java.lang.Thread.run(Thread.java:745)
>>>> Caused by: org.apache.maven.plugin.MojoFailureException: There are test
>>>> failures.
>>>> 
>>>> Please refer to
>>>> /root/nifi/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/target/surefire-reports
>>>> for the individual test results.
>>>>   at
>>>> org.apache.maven.plugin.surefire.SurefireHelper.reportExecution(SurefireHelper.java:82)
>>>>   at
>>>> org.apache.maven.plugin.surefire.SurefirePlugin.handl

Re: nifi fails build on CentOS 7

[Jeff]

yes, I’m running as root

> On Dec 7, 2015, at 4:05 PM, Joe Witt  wrote:
> 
> we now understand the TestPutHDFS issue.
> https://issues.apache.org/jira/browse/NIFI-1267
> 
> The test assumes that a user would not be able to write to a file
> unless it has write perms.  However, root can do that.  So users
> building as root would see this.
> 
> Thanks
> Joe
> 
> On Mon, Dec 7, 2015 at 4:45 PM, Joe Witt  wrote:
>> jeff - out of curiosity can you confirm you're running as root on that
>> VM when you build?
>> 
>> On Mon, Dec 7, 2015 at 4:42 PM, Jeff  wrote:
>>> Hi Joe,
>>> 
>>> You are welcome.  Below is what I’m seeing
>>> 
>>> ---
>>> Test set: org.apache.nifi.processors.hadoop.PutHDFSTest
>>> ---
>>> Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.949 sec 
>>> <<< FAILURE! - in org.apache.nifi.processors.hadoop.PutHDFSTest
>>> testPutFileWithException(org.apache.nifi.processors.hadoop.PutHDFSTest)  
>>> Time elapsed: 0.532 sec  <<< FAILURE!
>>> java.lang.AssertionError: null
>>>at org.junit.Assert.fail(Assert.java:86)
>>>at org.junit.Assert.assertTrue(Assert.java:41)
>>>at org.junit.Assert.assertFalse(Assert.java:64)
>>>at org.junit.Assert.assertFalse(Assert.java:74)
>>>at 
>>> org.apache.nifi.processors.hadoop.PutHDFSTest.testPutFileWithException(PutHDFSTest.java:230)
>>> 
>>>> On Dec 7, 2015, at 3:36 PM, Joe Witt  wrote:
>>>> 
>>>> thanks for reporting this Jeff and your timing could not have been better!
>>>> 
>>>> It's happening for Gilman as well.
>>>> 
>>>> Joe
>>>> 
>>>> On Mon, Dec 7, 2015 at 4:31 PM, Jeff  wrote:
>>>>> Yes, CentOS 7
>>>>> 
>>>>> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.703 sec 
>>>>> - in org.apache.nifi.processors.hadoop.TestListHDFS
>>>>> 
>>>>> Results :
>>>>> 
>>>>> Failed tests:
>>>>> PutHDFSTest.testPutFileWithException:230 null
>>>>> 
>>>>> 
>>>>> 
>>>>> Tests run: 21, Failures: 1, Errors: 0, Skipped: 0
>>>>> 
>>>>> 
>>>>> 
>>>>>> On Dec 7, 2015, at 3:28 PM, Matt Gilman  wrote:
>>>>>> 
>>>>>> Jeff,
>>>>>> 
>>>>>> Are you still running on CentOS 7? Can you provide more details about the
>>>>>> unit test is failing? Thanks!
>>>>>> 
>>>>>> Matt
>>>>>> 
>>>>>> On Mon, Dec 7, 2015 at 4:24 PM, Jeff  wrote:
>>>>>> 
>>>>>>> I just cloned Nifi but I’m still seeing these errors when I do mvn -e -X
>>>>>>> -T 2.0C clean install
>>>>>>> 
>>>>>>> [ERROR] Failed to execute goal
>>>>>>> org.apache.maven.plugins:maven-surefire-plugin:2.18:test (default-test) 
>>>>>>> on
>>>>>>> project nifi-hdfs-processors: There are test failures.
>>>>>>> [ERROR]
>>>>>>> [ERROR] Please refer to
>>>>>>> /root/nifi/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/target/surefire-reports
>>>>>>> for the individual test results.
>>>>>>> [ERROR] -> [Help 1]
>>>>>>> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to 
>>>>>>> execute
>>>>>>> goal org.apache.maven.plugins:maven-surefire-plugin:2.18:test
>>>>>>> (default-test) on project nifi-hdfs-processors: There are test failures.
>>>>>>> 
>>>>>>> Please refer to
>>>>>>> /root/nifi/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/target/surefire-reports
>>>>>>> for the individual test results.
>>>>>>>  at
>>>>>>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
>>>>>>>  at
>>>>>>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>>>>>>>  at
>>>>>>> org.apache.maven.lifecycle.internal.M

Re: nifi fails build on CentOS 7

[Jeff]

I created a nifi user on my VM, moved the /root/nifi folder to / and changed 
the owner of /nifi to nifi:nifi recursively

I executed mvn -e -X -T 2.0C clean install as the nifi user

Below is the error;

Downloaded: 
https://repo1.maven.org/maven2/com/yammer/metrics/metrics-ganglia/2.2.0/metrics-ganglia-2.2.0.jar
 (12 KB at 3.8 KB/sec)
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-clean-plugin:2.5:clean (default-clean) on 
project nifi-hdfs-processors: Failed to clean project: Failed to delete 
/nifi/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/target/testPutFileWrongPermissions
 -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal 
org.apache.maven.plugins:maven-clean-plugin:2.5:clean (default-clean) on 
project nifi-hdfs-processors: Failed to clean project: Failed to delete 
/nifi/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/target/testPutFileWrongPermissions
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:216)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at 
org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:189)
at 
org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:185)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.maven.plugin.MojoExecutionException: Failed to clean 
project: Failed to delete 
/nifi/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/target/testPutFileWrongPermissions
at org.apache.maven.plugin.clean.CleanMojo.execute(CleanMojo.java:215)
at 
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
... 11 more
Caused by: java.io.IOException: Failed to delete 
/nifi/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/target/testPutFileWrongPermissions
at org.apache.maven.plugin.clean.Cleaner.delete(Cleaner.java:249)
at org.apache.maven.plugin.clean.Cleaner.delete(Cleaner.java:191)
at org.apache.maven.plugin.clean.Cleaner.delete(Cleaner.java:158)
at org.apache.maven.plugin.clean.Cleaner.delete(Cleaner.java:117)
at org.apache.maven.plugin.clean.CleanMojo.execute(CleanMojo.java:193)
... 13 more
[ERROR] 
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :nifi-hdfs-processors
[DEBUG] incrementalBuildHelper#afterRebuildExecution
[INFO] 
[INFO] --- maven-resources-plugin:2.7:testResources (default-testResources) @ 
nifi-standard-prioritizers ---
[DEBUG] Configuring mojo 
org.apache.maven.plugins:maven-resources-plugin:2.7:testResources from plugin 
realm ClassRealm[plugin>org.apache.maven.plugins:maven-resources-plugin:2.7, 
parent: sun.misc.Launcher$AppClassLoader@70dea4e]
Downloaded: 
https://repo1.maven.org/maven2/com/yammer/metrics/metrics-core/2.2.0/metrics-core-2.2.0.jar
 (81 KB at 26.7 KB/sec)

> On Dec 7, 2015, at 4:05 PM, Joe Witt  wrote:
> 
> we now understand the TestPutHDFS issue.
> https://issues.apache.org/jira/browse/NIFI-1267
> 
> The test assumes that a user would not be able to write to a file
> unless it has write perms.  However, root can do that.  So users
> building as root would see this.
> 
> Thanks
> Joe
> 
> On Mon, Dec 7, 2015 at 4:45 PM, Joe Witt  wrote:
>> jeff - out of curiosity can you confirm you're running as root on that
>> VM when you build?
>> 
>> On Mon, Dec 7, 2015 at 4:42 PM, Jeff  wrote:
>>> Hi Joe,
>>> 
>>> You are welcome.  Below is what I’m seeing
>>> 
>>> ---
>>> Test set: org.apache.nifi.processors.hadoop.PutHDFSTest
>>> ---
>>> Tests run: 3, Failures: 1

Re: [VOTE] Release Apache NiFi 0.6.1 (RC2)

[Jeff]

+1 (non-binding)

Overall, the build worked as expected on my Windows 7 desktop.  I went
through the release guide and things looked good. NiFi starts up from the
convenience binary and works as expected.

Output from mvn -version infoon my Win7 desktop:

Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
2015-11-10T11:41:47-05:00)
Maven home: C:\Users\Jeff\nifi-dev\apache-maven-3.3.9\bin\..
Java version: 1.8.0_77, vendor: Oracle Corporation
Java home: c:\Progra~1\Java\jdk1.8.0_77\jre
Default locale: en_US, platform encoding: Cp1252
OS name: "windows 7", version: "6.1", arch: "amd64", family: "dos"

On my MBP laptop (from 2009/2010), there were many unit test failures with
the dealing with the HTTP-related processors.  During the weekend I'll
investigate the unit tests to see if I can figure out why the build fails
in this particular environment.

Output from mvn -version on the MBP:

Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
2015-11-10T11:41:47-05:00)
Maven home: /usr/local/Cellar/maven/3.3.9/libexec
Java version: 1.8.0, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.9.5", arch: "x86_64", family: "mac"

The *HTTP tests are failing

> > > >> On Apr 12, 2016, at 7:47 PM, Joe Witt  wrote:
> > > >>
> > > >> Hello Apache NiFi Community,
> > > >>
> > > >> I am pleased to be calling this vote for the source release of
> Apache
> > > >> NiFi 0.6.1.
> > > >>
> > > >> The source zip, including signatures, digests, etc. can be found at:
> > > >> https://dist.apache.org/repos/dist/dev/nifi/nifi-0.6.1/
> > > >>
> > > >> The Git tag is nifi-0.6.1-RC2
> > > >> The Git commit hash is 1a67b4de2e504bbe1a0cdbd6cccd949f997a5ad5
> > > >> *
> > >
> >
>
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=1a67b4de2e504bbe1a0cdbd6cccd949f997a5ad5
> > > >> *
> > >
> >
>
> https://github.com/apache/nifi/commit/1a67b4de2e504bbe1a0cdbd6cccd949f997a5ad5
> > > >>
> > > >> Checksums of nifi-0.6.1-source-release.zip:
> > > >> MD5: 5bb2b80e0384f89e6055ad4b0dd45294
> > > >> SHA1: b262664ed077f28623866d2a1090a4034dc3c04a
> > > >>
> > > >> Release artifacts are signed with the following key:
> > > >> https://people.apache.org/keys/committer/joewitt.asc
> > > >>
> > > >> KEYS file available here:
> > > >> https://dist.apache.org/repos/dist/release/nifi/KEYS
> > > >>
> > > >> 13 issues were closed/resolved for this release:
> > > >>
> > >
> >
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12335496
> > > >> Release note highlights can be found here:
> > > >>
> > >
> >
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.6.1
> > > >>
> > > >> The vote will be open for 72 hours.
> > > >> Please download the release candidate and evaluate the necessary
> items
> > > >> including checking hashes, signatures, build from source, and test.
> > Then
> > > >> please vote:
> > > >>
> > > >> [ ] +1 Release this package as nifi-0.6.1
> > > >> [ ] +0 no opinion
> > > >> [ ] -1 Do not release this package because...
> > > >>
> > > >> Thanks!
> > >
> > >
> >
>

Re: [VOTE] Release Apache NiFi 0.6.1 (RC2)

[Jeff]

erator.java:134)

at
org.apache.http.impl.conn.BasicHttpClientConnectionManager.connect(BasicHttpClientConnectionManager.java:338)

at
org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)

at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)

at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)

at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)

at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)

at org.apache.nifi.processors.standard.GetHTTP.onTrigger(GetHTTP.java:433)
at
org.apache.nifi.util.StandardProcessorTestRunner$RunProcessor.call(StandardProcessorTestRunner.java:288)

at
org.apache.nifi.util.StandardProcessorTestRunner$RunProcessor.call(StandardProcessorTestRunner.java:282)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:744)

On Sun, Apr 17, 2016 at 8:02 PM Joe Skora  wrote:

> Jeff,
>
> I have an early 2011 MBP, i7 2.0GHz that struggles to get through the HTTP
> tests, are yours failing with timeout errors?
>
> Mine frequently succeed if re-run, but not always.  Are your failures
> persistent?  Is this new, or have they always failed on that system?
>
> Regards,
> Joe
>
> On Fri, Apr 15, 2016 at 9:32 PM, Jeff  wrote:
>
> > +1 (non-binding)
> >
> > Overall, the build worked as expected on my Windows 7 desktop.  I went
> > through the release guide and things looked good. NiFi starts up from the
> > convenience binary and works as expected.
> >
> > Output from mvn -version infoon my Win7 desktop:
> >
> > Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
> > 2015-11-10T11:41:47-05:00)
> > Maven home: C:\Users\Jeff\nifi-dev\apache-maven-3.3.9\bin\..
> > Java version: 1.8.0_77, vendor: Oracle Corporation
> > Java home: c:\Progra~1\Java\jdk1.8.0_77\jre
> > Default locale: en_US, platform encoding: Cp1252
> > OS name: "windows 7", version: "6.1", arch: "amd64", family: "dos"
> >
> > On my MBP laptop (from 2009/2010), there were many unit test failures
> with
> > the dealing with the HTTP-related processors.  During the weekend I'll
> > investigate the unit tests to see if I can figure out why the build fails
> > in this particular environment.
> >
> > Output from mvn -version on the MBP:
> >
> > Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
> > 2015-11-10T11:41:47-05:00)
> > Maven home: /usr/local/Cellar/maven/3.3.9/libexec
> > Java version: 1.8.0, vendor: Oracle Corporation
> > Java home:
> /Library/Java/JavaVirtualMachines/jdk1.8.0.jdk/Contents/Home/jre
> > Default locale: en_US, platform encoding: UTF-8
> > OS name: "mac os x", version: "10.9.5", arch: "x86_64", family: "mac"
> >
> > The *HTTP tests are failing
> >
> > > > > >> On Apr 12, 2016, at 7:47 PM, Joe Witt 
> wrote:
> > > > > >>
> > > > > >> Hello Apache NiFi Community,
> > > > > >>
> > > > > >> I am pleased to be calling this vote for the source release of
> > > Apache
> > > > > >> NiFi 0.6.1.
> > > > > >>
> > > > > >> The source zip, including signatures, digests, etc. can be found
> > at:
> > > > > >> https://dist.apache.org/repos/dist/dev/nifi/nifi-0.6.1/
> > > > > >>
> > > > > >> The Git tag is nifi-0.6.1-RC2
> > > > > >> The Git commit hash is 1a67b4de2e504bbe1a0cdbd6cccd949f997a5ad5
> > > > > >> *
> > > > >
> > > >
> > >
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=1a67b4de2e504bbe1a0cdbd6cccd949f997a5ad5
> > > > > >> *
> > > > >
> > > >
> > >
> > >
> >
> https://github.com/apache/nifi/commit/1a67b4de2e504bbe1a0cdbd6cccd949f997a5ad5
> > > > > >>
> > > > > >> Checksums of nifi-0.6.1-source-release.zip:
> > > > > >> MD5: 5bb2b80e0384f89e6055ad4b0dd45294
> > > > > >> SHA1: b262664ed077

Re: [VOTE] Establish Registry, a sub-project of Apache NiFi

[Jeff]

+1 (non-binding)

On Fri, Feb 10, 2017 at 10:43 PM Brandon DeVries  wrote:

> +1 binding
> On Fri, Feb 10, 2017 at 6:36 PM Andre  wrote:
>
> > +1 binding
> >
> > On Sat, Feb 11, 2017 at 3:40 AM, Bryan Bende  wrote:
> >
> > > All,
> > >
> > > Following a solid discussion for the past few days [1] regarding the
> > > establishment of Registry as a sub-project of Apache NiFi, I'd like to
> > > call a formal vote to record this important community decision and
> > > establish consensus.
> > >
> > > The scope of this project is to define APIs for interacting with
> > > resources that one or more NiFi instances may be interested in, such
> > > as a flow registry for versioned flows, an extension registry for
> > > extensions, and possibly other configuration resources in the future.
> > > In addition, this project will provide reference implementations of
> > > these registries, with the goal of allowing the community to build a
> > > diverse set of implementations, such as a Git provider for versioned
> > > flows, or a bintray provider for an extension registry.
> > >
> > > I am a +1 and looking forward to the future work in this area.
> > >
> > > The vote will be open for 72 hours and be a majority rule vote.
> > >
> > > [ ] +1 Establish Registry, a subproject of Apache NiFi
> > > [ ]   0 Do not care
> > > [ ]  -1 Do not establish Registry, a subproject of Apache NiFi
> > >
> > > Thanks,
> > >
> > > Bryan
> > >
> > > [1]
> > http://mail-archives.apache.org/mod_mbox/nifi-dev/201702.mbox/%3CCALo_
> > > M19euo2LLy0PVWmE70FzeLhQRcCtX6TC%3DqoiBVfn4zFQMA%40mail.gmail.com%3E
> > >
> >
>

Re: Dealing with cluster errors

[Jeff]

Hello Joe,

What is the disk utilization on the nodes of your cluster while you're
having issues with using the UI?

I have done some testing under heavy disk utilization and have had to
increase the timeout values for cluster communication to prevent
replication requests from timing out.  Does your flow use Site-to-Site?

On Mon, Feb 13, 2017 at 11:43 AM Joe Gresock  wrote:

> "Can you tell us more about the processors using cluster scoped state and
> what the rates are through them?"
>
> In this case it's probably not relevant, because I have that processor
> stopped.  However, it's a custom MongoDB processor that stores the last
> mongo ID in the cluster scoped state, to enable scrolling through mongo
> results.  When it's enabled, it updates the state about  500 times /
> minute.
>
> Some other observations, though.. I've been able to manually throttle the
> data rate by slowly enabling more processors, while reducing their schedule
> thread count.  So far I've noticed that the issue is more likely to occur
> when the CPUs are maxed out for a while, though that's not particularly
> surprising.  I've noticed that prior to each time the console becomes
> unreachable, I tend to start seeing ThreadPoolRequestReplicator exceptions
> in the logs.
>
> I have also noticed that as my cluster has been draining flow files
> (started at around 3 million queued up), it's taking longer and longer to
> get into the bad state.  Not sure if this is related though, or if I've
> just lightened the CPU load by decreasing the scheduled threads.
>
> On Mon, Feb 13, 2017 at 4:25 PM, Joe Witt  wrote:
>
> > Joe
> >
> > Can you tell us more about the processors using cluster scoped state
> > and what the rates are through them?
> >
> > I could envision us putting too much strain on zk in some cases.
> >
> > Thanks
> > Joe
> >
> > On Mon, Feb 13, 2017 at 10:51 AM, Joe Gresock 
> wrote:
> > > I was able to externalize my zookeeper quorum, which is now running on
> 3
> > > separate VMs.  I am able to bring up the nifi cluster when my data flow
> > is
> > > stopped, and I can tell the zk migration worked because I have some
> > > processors with cluster-scoped state.
> > >
> > > However, I am still having a hard time getting the console to stay up,
> > with
> > > the same error messages from my original post.
> > >
> > > I also noticed the following error that I was wondering about:
> > >
> > > ThreadPoolRequestReplicator: Cannot replicate request GET
> > > /nifi-api/site-to-site because there are 100 outstanding HTTP Requests
> > > already.  Request Counts per URI = {/nifi-api/site-to-site=100}.
> > >
> > > I'm wondering if this is the underlying problem, though I don't know
> why
> > it
> > > would happen only during a high data volume, because I am currently not
> > > using site-to-site when I let the data run.  I have several self-RPG
> > > connections in the flow, but they are not being actively used when I
> > > process the data at the moment.
> > >
> > > Interestingly, I am able to run a custom processor that stores records
> in
> > > MongoDB without issue, but as soon as I run a RouteOnAttribute
> processor
> > as
> > > well, the console goes down again.
> > >
> > > Any other thoughts?
> > >
> > > On Fri, Feb 10, 2017 at 1:29 PM, Andrew Grande 
> > wrote:
> > >
> > >> Joe,
> > >>
> > >> External ZK quorum would be my first move. And make sure those boxes
> > have
> > >> fast disks and no heavy load from other processes.
> > >>
> > >> Andrew
> > >>
> > >> On Fri, Feb 10, 2017, 7:23 AM Joe Gresock  wrote:
> > >>
> > >> > I should add that the flows on the individual nodes appear to be
> > >> processing
> > >> > the data just fine, and the solution I've found so far is to just
> wait
> > >> for
> > >> > the data to subside, after which point the console comes up
> > successfully.
> > >> > So, no complaint on the durability of the underlying data flows.
> It's
> > >> just
> > >> > problematic that I can't reliably make changes to the flow during
> high
> > >> > traffic periods.
> > >> >
> > >> > On Fri, Feb 10, 2017 at 12:00 PM, Joe Gresock 
> > >> wrote:
> > >> >
> > >> > > We have a 7-node cluster and we currently use the embedded
> > zookeepers
> > >> on
> > >> > 3
> > >> > > of the nodes.  I've noticed that when we have a high volume in our
> > flow
> > >> > > (which is causing the CPU to be hit pretty hard), I have a really
> > hard
> > >> > time
> > >> > > getting the console page to come up, as it cycles through the
> > following
> > >> > > error messages when I relolad the page:
> > >> > >
> > >> > >
> > >> > >- An unexpected error has occurred.  Please check the logs.
> > (there
> > >> is
> > >> > >never any error in the logs for this one)
> > >> > >- Could not replicate request to  because the node is
> > not
> > >> > >connected   (this is never the current host I'm trying to hit,
> > which
> > >> > makes
> > >> > >the error text feel a bit irrelevant to the user.  i.e., "I
> > wasn't
> > >> > tryi

Re: Dealing with cluster errors

[Jeff]

Joe,

Some settings to try if these issues occur again:

In nifi.properties:
nifi.cluster.node.read.timeout=30 sec

In zookeeper.properties:
tickTime=5000

Try switching your RemoteGroup settings from HTTP to RAW, and set the
Communications Timeout to something like 1m.

On Mon, Feb 13, 2017 at 12:21 PM Joe Gresock  wrote:

> The disk utilization is currently 90-95% used by system and user, and
> iowait is very low.  We do use site-to-site.
>
> Interestingly, I can no longer replicate the problem, which is good but
> puzzling.  Since the problem first started, I have externalized the ZK
> quorum and decreased the scheduled threads for some processors.
>
> On Mon, Feb 13, 2017 at 5:15 PM, Jeff  wrote:
>
> > Hello Joe,
> >
> > What is the disk utilization on the nodes of your cluster while you're
> > having issues with using the UI?
> >
> > I have done some testing under heavy disk utilization and have had to
> > increase the timeout values for cluster communication to prevent
> > replication requests from timing out.  Does your flow use Site-to-Site?
> >
> > On Mon, Feb 13, 2017 at 11:43 AM Joe Gresock  wrote:
> >
> > > "Can you tell us more about the processors using cluster scoped state
> and
> > > what the rates are through them?"
> > >
> > > In this case it's probably not relevant, because I have that processor
> > > stopped.  However, it's a custom MongoDB processor that stores the last
> > > mongo ID in the cluster scoped state, to enable scrolling through mongo
> > > results.  When it's enabled, it updates the state about  500 times /
> > > minute.
> > >
> > > Some other observations, though.. I've been able to manually throttle
> the
> > > data rate by slowly enabling more processors, while reducing their
> > schedule
> > > thread count.  So far I've noticed that the issue is more likely to
> occur
> > > when the CPUs are maxed out for a while, though that's not particularly
> > > surprising.  I've noticed that prior to each time the console becomes
> > > unreachable, I tend to start seeing ThreadPoolRequestReplicator
> > exceptions
> > > in the logs.
> > >
> > > I have also noticed that as my cluster has been draining flow files
> > > (started at around 3 million queued up), it's taking longer and longer
> to
> > > get into the bad state.  Not sure if this is related though, or if I've
> > > just lightened the CPU load by decreasing the scheduled threads.
> > >
> > > On Mon, Feb 13, 2017 at 4:25 PM, Joe Witt  wrote:
> > >
> > > > Joe
> > > >
> > > > Can you tell us more about the processors using cluster scoped state
> > > > and what the rates are through them?
> > > >
> > > > I could envision us putting too much strain on zk in some cases.
> > > >
> > > > Thanks
> > > > Joe
> > > >
> > > > On Mon, Feb 13, 2017 at 10:51 AM, Joe Gresock 
> > > wrote:
> > > > > I was able to externalize my zookeeper quorum, which is now running
> > on
> > > 3
> > > > > separate VMs.  I am able to bring up the nifi cluster when my data
> > flow
> > > > is
> > > > > stopped, and I can tell the zk migration worked because I have some
> > > > > processors with cluster-scoped state.
> > > > >
> > > > > However, I am still having a hard time getting the console to stay
> > up,
> > > > with
> > > > > the same error messages from my original post.
> > > > >
> > > > > I also noticed the following error that I was wondering about:
> > > > >
> > > > > ThreadPoolRequestReplicator: Cannot replicate request GET
> > > > > /nifi-api/site-to-site because there are 100 outstanding HTTP
> > Requests
> > > > > already.  Request Counts per URI = {/nifi-api/site-to-site=100}.
> > > > >
> > > > > I'm wondering if this is the underlying problem, though I don't
> know
> > > why
> > > > it
> > > > > would happen only during a high data volume, because I am currently
> > not
> > > > > using site-to-site when I let the data run.  I have several
> self-RPG
> > > > > connections in the flow, but they are not being actively used when
> I
> > > > > process the data at the moment.
> > > > >
> > > > > Interestingl

Re: Compress a folder with all its files

[Jeff]

Just wanted to chime in here.  ListFile does not add fragment attributes,
which has already been pointed out in this thread, but there's a JIRA to
add this capability [1], though it has not been worked on yet.

This JIRA would allow for MergeContent to be able to create an archive of
all the files listed from a directory at the point at which the listing was
made, ie., a discrete listing.

[1] https://issues.apache.org/jira/browse/NIFI-2607

On Tue, Feb 7, 2017 at 6:39 PM teox78  wrote:

> Hi Mark,
>  I need to do something very similar and though of changing ListFile
> processor in order to add attributes as you suggested. My question is : in
> order to do this should I copy ListFile source code , make needed changes
> and save it as a new class ? Then package it as a NAR and add it to lib
> folder 
>
> Thank you
> Matteo
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Compress-a-folder-with-all-its-files-tp12186p14611.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: [ANNOUNCE] New Apache NiFi Committer Jeff Storck

[Jeff]

Thanks, all!  It's an honor to be a part of this team!

- Jeff

On Tue, Feb 21, 2017 at 7:55 PM Scott Aslan  wrote:

> Congrats Jeff! Well deserved!
>
> On Tue, Feb 21, 2017 at 6:36 PM, Koji Kawamura 
> wrote:
>
> > Congratulations Jeff!
> >
> > On Wed, Feb 22, 2017 at 7:52 AM, Andre  wrote:
> > > Welcome aboard Jeff! Well deserved
> > >
> > > On Wed, Feb 22, 2017 at 6:41 AM, Aldrin Piri 
> wrote:
> > >
> > >> On behalf of the Apache NiFI PMC, I am very pleased to announce that
> > Jeff
> > >> Storck has accepted the PMC's invitation to become a committer on the
> > >> Apache NiFi project. We greatly appreciate all of Jeff's hard work and
> > >> generous contributions and look forward to continued involvement in
> the
> > >> project.
> > >>
> > >> Jeff's contributions include significant efforts toward upgrade and
> > >> migration processes inclusive of flow layout when upgrading from 0.x
> to
> > 1.x
> > >> and the ZooKeeper migration toolkit.
> > >>
> > >> Congrats, Jeff!
> > >>
> >
>
>
>
> --
> *Scott Aslan = new WebDeveloper(*
> *{"location": {"city": "Saint Cloud","state": "FL",
> "zip": "34771"},"contact": {"email":
> "scottyas...@gmail.com ","linkedin":
> "http://www.linkedin.com/in/scottyaslan
> <http://www.linkedin.com/in/scottaslan>"}});*
>

Re: [ANNOUNCE] New Apache NiFi PMC Member - James Wing

[Jeff]

Congrats, James!

On Wed, Feb 22, 2017 at 10:13 AM Oleg Zhurakousky <
ozhurakou...@hortonworks.com> wrote:

> Awesome! Well earned! Congrats!
>
> > On Feb 22, 2017, at 10:09 AM, Joe Witt  wrote:
> >
> > congrats james and thanks for your efforts!
> >
> > On Wed, Feb 22, 2017 at 10:03 AM, Joe Skora  wrote:
> >> Congrats James!
> >>
> >> On Wed, Feb 22, 2017 at 9:58 AM, Aldrin Piri  wrote:
> >>
> >>> Team,
> >>>
> >>> On behalf of the Apache NiFi PMC, I am pleased to announce that James
> Wing
> >>> has accepted the PMC's invitation to join the Apache NiFi PMC.  We
> >>> greatly appreciate all of Joe's hard work and generous contributions to
> >>> the project. We look forward to his continued involvement in the
> project.
> >>>
> >>> James started out contributing in January 2016 with review and
> assistance
> >>> on all things AWS. After receiving committer status, James embraced the
> >>> role and continued to assist in fostering and growing the community.
> Beyond
> >>> code contributions, James is active in the community lists, votes,
> reviews
> >>> and external sites helping solve questions about NiFi.
> >>>
> >>> Please join us in congratulating and welcoming James to the Apache NiFi
> >>> PMC.
> >>>
> >>> Congratulations and welcome, James!
> >>>
> >
>
>

Re: Zookeeper issues at initial Cluster startup

[Jeff]

Hello Mark,

Sorry to hear that you're having issues with getting your cluster up and
running.  Could you provide the content of your nifi.properties file?
Also, please check the Admin guide for ZK setup [1], particularly the Flow
Election and Basic Cluster Setup sections.

By default, nifi.properties uses a 5-minute election duration to elect the
primary node.  However, it does not have a default number of candidates for
the election, so typically it will take 5 minutes for that election process
when you have a 3-node cluster.  You could try
setting nifi.cluster.flow.election.max.candidates to 3, and restart the
cluster, but based on the errors you're seeing, I think there may be some
other issues.

Some key properties to check:

nifi.properties:
nifi.state.management.embedded.zookeeper.start (true for embedded ZK, false
or blank if you're using an external ZK)
nifi.zookeeper.connect.string (set to the connect string for your ZK
quorum, regardless of embedded or external ZK, e.g.
host1:2181,host2:2181,host3:2181)

zookeeper.properties:
server.1 (server.1 through server.N, should be set to the hostname:port of
each ZK server in your cluster, regardless of embedded or external ZK)

state-management.xml, under cluster-provider element:
 (set to the connect string to
access your ZK quorum, used by processors to store cluster-based state)

[1]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#clustering

On Tue, Feb 28, 2017 at 12:56 PM Mark Bean  wrote:

> I am attempting to setup a new Cluster with 3 Nodes initially. Each node is
> reporting zookeeper/curator errors, and the Cluster is not able to connect
> the Nodes. The error is reported many times per second and is continuous on
> all Nodes:
>
> 2017-02-28 14:22:53,515 ERROR [Curator-Framework-0]
> o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java.99)
> ~[zookeeper-3.4.6.jar:3.4.6-1569965]
> at
>
> org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728)
> [curator-framework-2.11.0.jar:na]
> at
>
> org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:857)
> [curator-framework-2.11.0.jar:na]
> at
>
> org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809)
> [curator-framework-2.11.0.jar:na]
> at
>
> org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)
> [curator-framework-2.11.0.jar:na]
> at
>
> org.apache.curator.framework.imps.CuratorFrameworkImpl.$4.call(CuratorFrameworkImpl.java:267)
> [curator-framework-2.11.0.jar:na]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_121]
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_121]
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [na:1.8.0_121]
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_121]
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_121]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> 2017-02-28 14:22:53,516 ERROR [Curator-Framework-0]
> o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up
> org.apache.curator.CuratorConnectionLossException: KeeperErrorCode =
> ConnectionLoss
> at
>
> org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFramworkImpl.java:838)
> [curator-framework-2.11.0.jar:na]
> at
>
> org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809)
> [curator-framework-2.11.0.jar:na]
> at
>
> org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)
> [curator-framework-2.11.0.jar:na]
> at
>
> org.apache.curator.framework.imps.CuratorFrameworkImpl.$4.call(CuratorFrameworkImpl.java:267)
> [curator-framework-2.11.0.jar:na]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_121]
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_121]
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [na:1.8.0_121]
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_121]
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_121]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
>
> While the above message was repeating in the log on one of the Nodes,
> another Node's log

Re: Zookeeper issues at initial Cluster startup

[Jeff]

Mark,

In my original response, I said that in zookeeper.propertiers, the server.N
properties should be set to the host:port of your ZK server, and that was
pretty ambiguous.  It should not be set to the same port as clientPort.

As Bryan mentioned, with the default clientPort set to 2181, typically the
server.N properties are set to hostname:2888:3888.  In your case, you might
want to try something like the following, as long as these ports are not
currently in use:
server.1=:2888:3888
server.2=:2888:3888
server.3=:2888:3888

Also, your settings for leader elections:
nifi.cluster.flow.election.max.wait.time=5 mins
nifi.cluster.flow.election.max.candidates=201

This will wait for 201 election candidates to connect, or 5 minutes.  You
might want to set the max candidates to 3, since you have 3 nodes in your
cluster.

The contents of ./state/zookeeper look correct, you should be okay there.


On Tue, Feb 28, 2017 at 2:19 PM Bryan Bende  wrote:

> Mark,
>
> I am not totally sure, but there could be an issue with the ports in
> some of the connect strings.
>
> In zookeeper.properties there is an entry for clientPort which
> defaults to 2181, the value of this property is what should be
> referenced in nifi.zookeeper.connect.string and state-management.xml
> Connect String, so if you left it alone then:
>
> FQDN1:2181,FQDN2:2181,FQDN3:2181
>
> In the server entries in zookeeper.properties, I believe they should
> be referencing different ports. For example, when using the default
> clientPort=2181 the server entries are typically like:
>
> server.1=localhost:2888:3888
>
> From the ZooKeeper docs the definition for these two ports is:
>
> "There are two port numbers n. The first followers use to connect
> to the leader, and the second is for leader election. The leader
> election port is only necessary if electionAlg is 1, 2, or 3
> (default). If electionAlg is 0, then the second port is not necessary.
> If you want to test multiple servers on a single machine, then
> different ports can be used for each server."
>
> In your configs it looks like the clientPort and the first port in the
> server string are both 11001, so I think making those different should
> do the trick.
>
> -Bryan
>
>
> On Tue, Feb 28, 2017 at 1:58 PM, Mark Bean  wrote:
> > Relevant properties from nifi.properties:
> > nifi.state.management.provider.cluster=zk-provider
> > nifi.state.management.embedded.zookeeper.start=true
> >
> nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties
> > nifi.cluster.protocol.heartbeat.interval=5 sec
> > nifi.cluster.protocol.is.secure=true
> > ## Security properties verified; they work for https in non-cluster
> > configuration
> >
> > nifi.cluster.is.node=true
> > nifi.cluster.node.address=FQDN1
> > nifi.cluster.node.protocol.port=9445
> > nifi.cluster.node.protocol.threads=10
> > nifi.cluster.node.event.history.size=25
> > nifi.cluster.node.connection.timeout=5 sec
> > nifi.cluster.node.read.timeout=5 sec
> > nifi.cluster.firewall.file=
> > nifi.cluster.flow.election.max.wait.time=5 mins
> > nifi.cluster.flow.election.max.candidates=201
> >
> > nifi.zookeeper.connect.string=FQDN1:11001,FQDN2:11001,FQDN3:11001
> > nifi.zookeeper.connect.tiemout=3 secs
> > nifi.zookeeper.session.timeout=3 secs
> > nifi.zookeeper.root.node=/nifi/test-cluster
> >
> > zookeeper.properties all default except added these lines:
> > server.1=:11001:11000
> > server.2=:11001:11000
> > server.3=:11001:11000
> >
> > state-management.xml all default except the following in
> :
> > FQDN1:11001,FQDN2:11001,FQDN3:11001
> > /nifi/test-cluster
> >
> > Also, the ./state/zookeeper/myid consists of only "1", "2", or "3"
> > depending on the server within the cluster. Is this correct?
> >
> >
> > On Tue, Feb 28, 2017 at 1:24 PM, Jeff  wrote:
> >
> >> Hello Mark,
> >>
> >> Sorry to hear that you're having issues with getting your cluster up and
> >> running.  Could you provide the content of your nifi.properties file?
> >> Also, please check the Admin guide for ZK setup [1], particularly the
> Flow
> >> Election and Basic Cluster Setup sections.
> >>
> >> By default, nifi.properties uses a 5-minute election duration to elect
> the
> >> primary node.  However, it does not have a default number of candidates
> for
> >> the election, so typically it will take 5 minutes for that election
> process
> >> when you have a 3-node cluster.  You could try
> >> setting nifi.cluster.flow.election.max.candidates to 3,

Re: Zookeeper issues at initial Cluster startup

[Jeff]

Mark,
There's some copy/paste errors in my last response as well.  Sorry!
server.1=:2888:3888
server.2=:2888:3888
server.3=:2888:3888

On Tue, Feb 28, 2017 at 2:31 PM Jeff  wrote:

> Mark,
>
> In my original response, I said that in zookeeper.propertiers, the
> server.N properties should be set to the host:port of your ZK server, and
> that was pretty ambiguous.  It should not be set to the same port as
> clientPort.
>
> As Bryan mentioned, with the default clientPort set to 2181, typically the
> server.N properties are set to hostname:2888:3888.  In your case, you might
> want to try something like the following, as long as these ports are not
> currently in use:
> server.1=:2888:3888
> server.2=:2888:3888
> server.3=:2888:3888
>
> Also, your settings for leader elections:
> nifi.cluster.flow.election.max.wait.time=5 mins
> nifi.cluster.flow.election.max.candidates=201
>
> This will wait for 201 election candidates to connect, or 5 minutes.  You
> might want to set the max candidates to 3, since you have 3 nodes in your
> cluster.
>
> The contents of ./state/zookeeper look correct, you should be okay there.
>
>
> On Tue, Feb 28, 2017 at 2:19 PM Bryan Bende  wrote:
>
> Mark,
>
> I am not totally sure, but there could be an issue with the ports in
> some of the connect strings.
>
> In zookeeper.properties there is an entry for clientPort which
> defaults to 2181, the value of this property is what should be
> referenced in nifi.zookeeper.connect.string and state-management.xml
> Connect String, so if you left it alone then:
>
> FQDN1:2181,FQDN2:2181,FQDN3:2181
>
> In the server entries in zookeeper.properties, I believe they should
> be referencing different ports. For example, when using the default
> clientPort=2181 the server entries are typically like:
>
> server.1=localhost:2888:3888
>
> From the ZooKeeper docs the definition for these two ports is:
>
> "There are two port numbers n. The first followers use to connect
> to the leader, and the second is for leader election. The leader
> election port is only necessary if electionAlg is 1, 2, or 3
> (default). If electionAlg is 0, then the second port is not necessary.
> If you want to test multiple servers on a single machine, then
> different ports can be used for each server."
>
> In your configs it looks like the clientPort and the first port in the
> server string are both 11001, so I think making those different should
> do the trick.
>
> -Bryan
>
>
> On Tue, Feb 28, 2017 at 1:58 PM, Mark Bean  wrote:
> > Relevant properties from nifi.properties:
> > nifi.state.management.provider.cluster=zk-provider
> > nifi.state.management.embedded.zookeeper.start=true
> >
> nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties
> > nifi.cluster.protocol.heartbeat.interval=5 sec
> > nifi.cluster.protocol.is.secure=true
> > ## Security properties verified; they work for https in non-cluster
> > configuration
> >
> > nifi.cluster.is.node=true
> > nifi.cluster.node.address=FQDN1
> > nifi.cluster.node.protocol.port=9445
> > nifi.cluster.node.protocol.threads=10
> > nifi.cluster.node.event.history.size=25
> > nifi.cluster.node.connection.timeout=5 sec
> > nifi.cluster.node.read.timeout=5 sec
> > nifi.cluster.firewall.file=
> > nifi.cluster.flow.election.max.wait.time=5 mins
> > nifi.cluster.flow.election.max.candidates=201
> >
> > nifi.zookeeper.connect.string=FQDN1:11001,FQDN2:11001,FQDN3:11001
> > nifi.zookeeper.connect.tiemout=3 secs
> > nifi.zookeeper.session.timeout=3 secs
> > nifi.zookeeper.root.node=/nifi/test-cluster
> >
> > zookeeper.properties all default except added these lines:
> > server.1=:11001:11000
> > server.2=:11001:11000
> > server.3=:11001:11000
> >
> > state-management.xml all default except the following in
> :
> > FQDN1:11001,FQDN2:11001,FQDN3:11001
> > /nifi/test-cluster
> >
> > Also, the ./state/zookeeper/myid consists of only "1", "2", or "3"
> > depending on the server within the cluster. Is this correct?
> >
> >
> > On Tue, Feb 28, 2017 at 1:24 PM, Jeff  wrote:
> >
> >> Hello Mark,
> >>
> >> Sorry to hear that you're having issues with getting your cluster up and
> >> running.  Could you provide the content of your nifi.properties file?
> >> Also, please check the Admin guide for ZK setup [1], particularly the
> Flow
> >> Election and Basic Cluster Setup sections.
> >>
> >> By default, nifi.properties uses a 5-minute election duration to elect
> the
&

Re: [ANNOUNCE] New Apache NiFi PMC Member - Oleg Zhurakousky

[Jeff]

Congrats, Oleg!

On Tue, Apr 4, 2017 at 6:33 PM Andy LoPresto  wrote:

> Congratulations Oleg. Very excited for your continued excellent work on
> the project and help for everyone on the lists.
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Apr 4, 2017, at 3:22 PM, Joe Witt  wrote:
>
> Oleg
>
> Thanks for all you've done and congrats on joining the PMC!
>
> Thanks
> Joe
>
> On Apr 4, 2017 3:51 PM, "Tony Kurc"  wrote:
>
> Apache NiFi Community,
>
> On behalf of the Apache NiFi PMC, I am pleased to announce that Oleg
> Zhurakousky has accepted the PMC's invitation to join the Apache NiFi PMC.
> We greatly appreciate all of Oleg’s hard work and generous contributions to
> the project. We look forward to his continued involvement in the project.
>
> Oleg started out contributing in late 2015, diving in deep on framework
> issues and contributing exciting new capabilities. I had the pleasure of
> reviewing some of those early contributions and always found the dialog
> with Oleg on reviews and ideas to be fun and stimulating. Since accepting
> the PMC’s invitation to become a committer, he’s maintained a steady hand
> in reviewing new contributions, staying active contributing code and on the
> mailing lists, and becoming an ever more valuable member of our community.
>
>
> Please join us in congratulating and welcoming Oleg to the Apache NiFi PMC.
>
> Congratulations and welcome, Oleg!
>
>
>

Re: [ANNOUNCE] New Apache NiFi Committer Bin Qiu

[Jeff]

Congrats to you, Bin!

On Tue, Apr 4, 2017 at 6:17 PM Andrew Psaltis 
wrote:

> Congrats Bin!
>
> On Tue, Apr 4, 2017 at 4:17 PM, Tony Kurc  wrote:
>
> > Apache NiFi community,
> >
> >
> > On behalf of the Apache NiFI PMC, I am very pleased to announce that Bin
> > Qiu has accepted the PMC's invitation to become a committer on the Apache
> > NiFi project. We greatly appreciate all of Bin’s hard work and generous
> > contributions and look forward to continued involvement in the project.
> >
> > Bin has been one of the driving forces behind MiNiFi - C++, contributing
> a
> > massive amount in terms of code and helping with release verification and
> > always being responsive to the community.
> >
> >
> > Congrats, Bin!
> >
> >
> > - Tony
> >
>
>
>
> --
> Thanks,
> Andrew
>
> Subscribe to my book: Streaming Data 
> 
> twiiter: @itmdata 
>

Re: [Urgent] Injecting a Java Agent into Apache NiFi

[Jeff]

Michele,

I'm not a Windows shell programming expert by any means, but I think your
quotes might be throwing things off a bit.  I took a quick look at this
link [1], and it looks like you might want to try surrounding the
parameters passed to "cmd", not including the "/k", with double-quotes.
Something like:

cmd.exe /K ""%JAVA_EXE%" %JAVA_PARAMS% %DYNA_ARGS% %BOOTSTRAP_ACTION%"


[1] https://ss64.com/nt/syntax-esc.html



On Tue, Apr 11, 2017 at 4:55 PM Michele  wrote:

> Hello *NiFi Dev Team,*
>
> Good day!
>
> I have an inquiry regarding how Apache NiFi starts up on Windows.
>
> I need to use Dynatrace to monitor Apache NiFi application. Dynatrace is
> an application monitoring software that can monitor any application on any
> platform.
> https://community.dynatrace.com/community/pages/viewpage.action?title=Get+Started+with+Dynatrace&spaceKey=DOCDT62
>
>
> Specifically for NiFi which is Java-based (correct?), the first thing to
> do is to inject an agent in the start file of NiFi (run-nifi.bat) by
> placing below command:
>
> *-agentpath:"C:\Dynatrace
> 6.5\agent\lib64\dtagent.dll"=name=JavaApplication_Test,server=localhost:9998*
>
> But when I do so, I get a Windows batch command error when I run 
> *run-nifi.bat:
>**"Input line is too long"*
>
> Attached files:
>
> run-nifi-orig.txt is the original start file for NiFi for Windows
> run-nifi.bat.txt is the edited start file where the agent 
> (*-agentpath:"C:\Dynatrace
> 6.5\agent\lib64\dtagent.dll"=name=JavaApplication_Test,server=localhost:9998)
> *is inserted and set as DYNA_ARGS
>
>
> Looking forward to your response. Thank you very much in advance.
>
>
>
> Sincerely,
>
> Michele
>
>

Re: [DISCUSS] Component deprecation annotation

[Jeff]

I think an icon on the processor would be best to denote an upcoming
deprecation.  That leaves all colors available to the flow design, since
any color may have specific meaning to any current flow out there.

On Sat, Apr 29, 2017 at 3:58 PM Juan Sequeiros  wrote:

> In same train of though maybe make it default to some color other than
> white, instinct says red but that might be used to represent an important
> processor. So suggest maybe default beige?
>
>
> On Sat, Apr 29, 2017 at 8:20 AM Andy LoPresto 
> wrote:
>
> > I'd leave the graphics work/decisions to someone like Rob Moran, but I
> see
> > it as similar to the restricted components -- an icon on the processors
> on
> > the canvas and the dialog to add components, with additional explanation
> > (and maybe a target release for full deprecation) in the help notes.
> >
> > Andy LoPresto
> > alopre...@apache.org
> > alopresto.apa...@gmail.com
> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >
> > > On Apr 29, 2017, at 05:16, Andre  wrote:
> > >
> > > dev,
> > >
> > > In light of the some recent deprecations (ListenLumberjack) and some
> > > potential deprecation (PutJMS/GetJMS) I started some progress on how to
> > > document the depreciation of NiFi components using annotations.
> > >
> > > In the absence of any better name I called the annotation
> > > @DeprecationWarning (so not overlap with Java's @Deprecated)
> > >
> > > The suggested modifications are part of PR#1718 and I am keen to hear
> > your
> > > thoughts.
> > >
> > > While the documentation can be displayed as part of the usage, however
> I
> > > would imagine the frequency a user browses to the usage pages will
> reduce
> > > as her/him becomes familiar with the components.
> > >
> > > In light of this it may be a good idea to use  the web UI to highlight
> a
> > > processor has been deprecated.
> > >
> > > Would anyone suggest a way of doing so without disrupting the user
> > > experience?
> > >
> > > Cheers
> >
>

Re: nifi-extension-utils

[Jeff]

Thanks for doing this, Bryan.

As part of the PR for NIFI-1833 (Azure Storage Blob processors), I've moved
AbstractListProcessor (and some supporting classes) into
nifi-commons/nifi-processor-utils, and had to rebase for this change.  It
was a pretty simple process.  If anyone else needs help if/when they are
affected by this, I'd be happy to lend a hand.  The rebase pretty much did
all the heavy lifting, I just had to manually move the files to the new
location for nifi-processor-utils before doing the "git clean -df".

- Jeff

On Mon, May 1, 2017 at 5:25 PM Bryan Bende  wrote:

> All,
>
> As part of NIFI-3724 which was merged to master earlier today, I made
> some slight refactoring to the organization of some utility code and
> wanted to mention it in case anyone is wondering where things moved...
>
> Previously under nifi-commons we had nifi-processor-utils and
> nifi-hadoop-utils which had become places to put abstract processors
> and processor related code. This type of utility code is a bit
> different from the rest of nifi-commons which is completely
> independent of anything under nifi-nar-bundles.
>
> So I created nifi-extension-utils under nifi-nar-bundles to contain
> code like abstract processors and things that need to reference
> controller service APIs. The structure currently looks like the
> following:
>
> nifi-nar-bundles
> - nifi-extension-utils
>   - nifi-processor-utils
>   - nifi-hadoop-utils
>   - nifi-record-utils
> - nifi-standard-record-utils
> - nifi-avro-record-utils
> - nifi-hadoop-record-utils
>
> Going forward the goal would be for nifi-commons to be independent of
> anything in nifi-nar-bundles.
>
> After rebasing you may have the residual projects still under
> nifi-commons and will want to clean up appropriately. "git clean -dn"
> will dry run the clean and then "git clean -df" will delete it.
>
> Thanks,
>
> Bryan
>

Re: [VOTE] Release Apache NiFi 1.2.0 (RC2)

[Jeff]

+1 (non-binding)

Built and tested flows on Mac OS X El Capitan (maven 3.3.9, java 1.8.0-102,
with -T C2 and contrib-check) and Windows 7 (maven 3.3.9, java 1.8.0-77,
skipped the tests since the machine is so old and would've taken 32 hours
to build).

Great work everyone!  This is a great release!

On Mon, May 8, 2017 at 3:42 PM Joe Percivall  wrote:

> +1 (binding)
>
>
> Verified the hashes, commit id and tag. Built on Windows 8.1 and Mac
> 10.12.2. Mac built with no issues. Windows build ran into the issues
> identified here[1]. All appear to be OS compatibility issues with the unit
> tests and not problems with code itself (thus shouldn't affect the
> release).
>
> Also deployed to a long running dev centos7 docker instance and ran into
> this issue[2]. Does not appear to be important enough to hinder the
> release.
>
> Great job to everyone on this release! I'm looking forward to the community
> getting their hands on it.
>
>
> [1] https://issues.apache.org/jira/browse/NIFI-3840
> [2] https://issues.apache.org/jira/browse/NIFI-3839
>
> On Mon, May 8, 2017 at 2:56 PM, Michael Moser  wrote:
>
> > +1 (non-binding) Release this package as nifi-1.2.0
> >
> > There was definitely a lot of work that went into this release, many
> thanks
> > to all.
> >
> >
> > On Mon, May 8, 2017 at 2:23 PM, Mark Payne  wrote:
> >
> > > +1 (binding).
> > >
> > > Was able to complete full build w/ contrib check on OSX. Verified
> > > checksums and that a quick test of functionality
> > > looks good.
> > >
> > >
> > > > On May 5, 2017, at 10:07 PM, Bryan Bende  wrote:
> > > >
> > > > Hello,
> > > >
> > > > I am pleased to be calling this vote for the source release of Apache
> > > > NiFi nifi-1.2.0 (RC2).
> > > >
> > > > The source zip, including signatures, digests, etc. can be found at:
> > > >
> https://repository.apache.org/content/repositories/orgapachenifi-1104
> > > >
> > > > The Git tag is nifi-1.2.0-RC2
> > > > The Git commit ID is 3a605af8e0ac024fb0ba67262d49dab2727b2576
> > > > https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> > > 3a605af8e0ac024fb0ba67262d49dab2727b2576
> > > >
> > > > Checksums of nifi-1.2.0-source-release.zip:
> > > > MD5: 90e298a9e23a9dab65358daddd8b5990
> > > > SHA1: e138941f576bdb1dff17df6674c19ffae3ef6719
> > > > SHA256: c18398800c435dabff9f45ec55a450ca78c3c1aec222aa295c0e2057912f
> > eeb6
> > > >
> > > > Release artifacts are signed with the following key:
> > > > https://people.apache.org/keys/committer/bbende.asc
> > > >
> > > > KEYS file available here:
> > > > https://dist.apache.org/repos/dist/release/nifi/KEYS
> > > >
> > > > 381 issues were closed/resolved for this release:
> > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > > projectId=12316020&version=12338432
> > > >
> > > > Release note highlights can be found here:
> > > > https://cwiki.apache.org/confluence/display/NIFI/
> > > Release+Notes#ReleaseNotes-Version1.2.0
> > > >
> > > > The vote will be open for 72 hours.
> > > > Please download the release candidate and evaluate the necessary
> items
> > > > including checking hashes, signatures, build
> > > > from source, and test.  The please vote:
> > > >
> > > > [ ] +1 Release this package as nifi-1.2.0
> > > > [ ] +0 no opinion
> > > > [ ] -1 Do not release this package because because...
> > >
> > >
> >
>
>
>
> --
> *Joe Percivall*
> linkedin.com/in/Percivall
> e: jperciv...@apache.com
>

Re: Pulling out hair over slf4j...

[Jeff]

Tika, in the 1.0 tag, has slf4j-log4j12 [1] as a provided dependency,
version 1.5.6 (which I see gets brought in to your maven repo).  Later
versions of Tika are using maven-shade-plugin, but it doesn't look like 1.0
does.

Have you tried excluding that dependency from tika-app?

Oddly, tika-bundle uses a different version of slf4j-simple [2] for tests
(1.6.1).

[1] https://github.com/apache/tika/blob/1.0/tika-app/pom.xml#L53
[2] https://github.com/apache/tika/blob/1.0/tika-bundle/pom.xml#L99

On Tue, May 9, 2017 at 10:03 AM Russell Bateman 
wrote:

> Thanks, Andrew.I haven't tried that before (a new diagnostic tool in my
> quiver).
>
> I did it, scraped the IntelliJ IDEA test console, then split lines on
> colons and grep'd for /slf4j/ to get this:
>
> *master ~/notes $ fgrep slf4j debug-console.log *
> :/home/russ/.m2/repository/org/slf4j/slf4j-api/1.7.25/slf4j-api-1.7.25.jar
>
> :/home/russ/.m2/repository/org/slf4j/slf4j-simple/1.7.25/slf4j-simple-1.7.25.jar
>
> :/home/russ/.m2/repository/org/slf4j/jcl-over-slf4j/1.7.25/jcl-over-slf4j-1.7.25.jar
>
> :/home/russ/.m2/repository/org/slf4j/log4j-over-slf4j/1.7.25/log4j-over-slf4j-1.7.25.jar
>
> :/home/russ/.m2/repository/org/slf4j/jul-to-slf4j/1.7.25/jul-to-slf4j-1.7.25.jar
> [Loaded org.slf4j.LoggerFactory from
>
> file:/home/russ/.m2/repository/org/apache/tika/tika-app/1.0/tika-app-1.0.jar]
> [Loaded org.slf4j.ILoggerFactory from
>
> file:/home/russ/.m2/repository/org/apache/tika/tika-app/1.0/tika-app-1.0.jar]
> [Loaded org.slf4j.helpers.SubstituteLoggerFactory from
>
> file:/home/russ/.m2/repository/org/apache/tika/tika-app/1.0/tika-app-1.0.jar]
> [Loaded org.slf4j.Logger from
>
> file:/home/russ/.m2/repository/org/apache/tika/tika-app/1.0/tika-app-1.0.jar]
> [Loaded org.slf4j.spi.LoggerFactoryBinder from
>
> file:/home/russ/.m2/repository/org/apache/tika/tika-app/1.0/tika-app-1.0.jar]
> [Loaded org.slf4j.impl.StaticLoggerBinder from
>
> file:/home/russ/.m2/repository/com/perfectsearchcorp/medical-filter/195/medical-filter-195.jar]
> [Loaded org.slf4j.helpers.MessageFormatter from
>
> file:/home/russ/.m2/repository/org/apache/tika/tika-app/1.0/tika-app-1.0.jar]
> java.lang.AssertionError: java.lang.NoSuchMethodError:
>
> org.slf4j.helpers.MessageFormatter.arrayFormat(Ljava/lang/String;[Ljava/lang/Object;)Lorg/slf4j/helpers/FormattingTuple;
> Caused by: java.lang.NoSuchMethodError:
>
> org.slf4j.helpers.MessageFormatter.arrayFormat(Ljava/lang/String;[Ljava/lang/Object;)Lorg/slf4j/helpers/FormattingTuple;
>
>
> I'm very concerned by Tika having hard-linked /slf4j/. Is that what's
> going on? I know from experience that I cannot use any other version of
> Tika.* Anyway, what does this tell you?
>
> I greatly appreciate your help and suggestions in this.
>
> Russ
>
> * I've been working on excerpting those bits of our code base that
> consume Tika to put them into a different project (and NAR). It's not
> been easy and I haven't finished yet doing it in the midst of pretty
> tight deadlines. My former colleague left things pretty interdependent
> in February when he left (and told me it was going to be a lot of work
> to separate what we call "legacy" processors out from the rest).
>
> On 05/08/2017 08:37 PM, Andrew Psaltis wrote:
> > Russell,
> > Sorry this is so much of a pain. I wonder if you can pass using the JVM
> > argument -verbose:class and then running the test can shed more light on
> > which JAR is pulling it in.
> >
> > On Mon, May 8, 2017 at 5:14 PM, Russell Bateman 
> > wrote:
> >
> >> I understand that no NiFi JAR comes with /slf4j/, but expects the
> >> consuming build to provide it. That's the assumption I've been going on
> and
> >> it's why in my top-level /pom.xml/, I've specified [1.7.25] so that no
> >> quibbling over version can occur and I specify these JARs in the root
> >> /pom.xml/ so that no "renegade" submodule calls the dependency for its
> own.
> >>
> >> The problem is that some version of slf4j-api-x.y.z.jar doesn't provide
> a
> >> MessageFormatter class containing an arrayFormat() method, right?
> Somehow,
> >> though, this impoverished version is the one bound to my code despite my
> >> marking every dependency with Andrew's exclusion statements?
> >>
> >> There is nothing giving this away; in IntelliJ IDEA, under External
> >> Libraries in the Project pane, I see only 1.7.25 of these libraries, no
> >> additional versions. I wonder if I should see more versions if in fact
> >> that's what's going on?
> >>
> >> I tried wiping /~/.m2/repository/org/slf4j/, after mvn clean compile of
> my
> >> project, I see all those versions have come back to
> >> /~/.m2/repository/org/slf4j/. So, at the root of my project, I created
> >> /hard-repository/, and stuffed it with the slf4j-relevant JARs arranged
> in
> >> their repository accoutrements (paths, files, everything), then added
> >>
> >> 
> >>
> >>  slf4j
> >>  slf4j repo
> >>
>  file://${project.build.directory}/hard-repository/maven_repo
> >>   

Re: [ANNOUNCE] New Apache NiFi PMC Member - Yolanda Davis

[Jeff]

Congrats Yolanda!

On Wed, May 17, 2017 at 10:43 PM Yolanda Davis 
wrote:

> Thank you everyone! It has been my pleasure to continue to contribute to
> and support this awesome community. Definitely looking forward to all the
> great things ahead!
>
> On Wed, May 17, 2017 at 10:32 PM, Koji Kawamura 
> wrote:
>
> > Congratulations Yolanda, well deserved! I have a great respect for
> > your excellent work.
> >
> > On Thu, May 18, 2017 at 11:15 AM, Andy LoPresto 
> > wrote:
> > > Congratulations Yolanda. Well deserved and very excited for your
> > continued
> > > excellent contributions to all aspects of the project.
> > >
> > > Andy LoPresto
> > > alopre...@apache.org
> > > alopresto.apa...@gmail.com
> > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> > >
> > > On May 17, 2017, at 7:14 PM, Joe Witt  wrote:
> > >
> > > Agree very much with Tony's excellent writeup.
> > >
> > > Congrats and thank you Yolanda!
> > >
> > > On Wed, May 17, 2017 at 10:11 PM, Tony Kurc  wrote:
> > >
> > > On behalf of the Apache NiFi PMC, I am pleased to announce that Yolanda
> > > Davis has accepted the PMC's invitation to join the Apache NiFi PMC.
> We
> > > greatly appreciate all of Yolanda's hard work and generous
> contributions
> > to
> > > the project. We look forward to continued involvement in the project.
> > >
> > > I know many people have interacted with Yolanda maybe at one of her
> > > conference talks, or had the pleasure of working with her on reviews,
> or
> > > used some of the features she's contributed and put a lot of work into
> > > making them well tested and easy to use. If so, I think you'd agree it
> is
> > > hard to interact with Yolanda and not get excited about not only NiFi
> but
> > > the entire Apache way. If not, you're in for a treat when you get the
> > > chance!
> > >
> > > Please join us in congratulating and welcoming Yolanda to the Apache
> NiFi
> > > PMC.
> > >
> > > Congratulations and welcome, Yolanda!
> > >
> > > -- Tony
> > >
> > >
> >
>
>
>
> --
> --
> yolanda.m.da...@gmail.com
> @YolandaMDavis
>

Re: unstable cluster

[Jeff]

Mark,

I did report a JIRA [1] for upgrading to 3.5.2 or 3.6.0 (just due to log4j
issues) once it's out and stable, There are issues with the way that ZK
refers to log4j classes in the code that cause issues for NiFi and our
Toolkit..  However there has been some back and forth [2] (in 3.4.0, which
doesn't fix the issue, but moves towards fixing it), [3], and [4] on the
changes being implemented in versions 3.5.2 and 3.6.0.  Also, it looks like
ZK 3.6.0 is headed toward using log4j 2 [5].

There are many components outside of NiFi that are still using ZK 3.4.6, so
it may be a while before we can move to 3.4.10. I don't currently know
anything about the forward compatibility of 3.4.6.  Are there
improvements/fixes in 3.4.10 which you need?

[1] https://issues.apache.org/jira/browse/NIFI-3067
[2] https://issues.apache.org/jira/browse/ZOOKEEPER-850
[3] https://issues.apache.org/jira/browse/ZOOKEEPER-1371
[4] https://issues.apache.org/jira/browse/ZOOKEEPER-2393
[5] https://issues.apache.org/jira/browse/ZOOKEEPER-2342

- Jeff

On Tue, May 30, 2017 at 8:15 AM Mark Bean  wrote:

> Updated to external ZooKeeper last Friday. Over the weekend, there are no
> reports of SUSPENDED or RECONNECTED.
>
> Are there plans to upgrade the embedded ZooKeeper to the latest version,
> 3.4.10?
>
> Thanks,
> Mark
>
> On Thu, May 25, 2017 at 11:56 AM, Joe Witt  wrote:
>
> > looked at a secured cluster and the send times are routinely at 100ms
> > similar to yours.  I think what i was flagging as potentially
> > interesting is not interesting at all.
> >
> > On Thu, May 25, 2017 at 11:34 AM, Joe Witt  wrote:
> > > Ok.  Well as a point of comparison i'm looking at heartbeat logs from
> > > another cluster and the times are consistently 1-3 millis for the
> > > send.  Yours above show 100+ms typical with one north of 900ms.  Not
> > > sure how relevant that is but something i noticed.
> > >
> > > On Thu, May 25, 2017 at 11:29 AM, Mark Bean 
> > wrote:
> > >> ping shows acceptably fast response time between servers,
> approximately
> > >> 0.100-0.150 ms
> > >>
> > >>
> > >> On Thu, May 25, 2017 at 11:13 AM, Joe Witt 
> wrote:
> > >>
> > >>> have you evaluated latency across the machines in your cluster?  I
> ask
> > >>> because 122ms is pretty long and 917ms is very long.  Are these nodes
> > >>> across a WAN link?
> > >>>
> > >>> On Thu, May 25, 2017 at 11:08 AM, Mark Bean 
> > wrote:
> > >>> > Update: now all 5 nodes, regardless of ZK server, are indicating
> > >>> SUSPENDED
> > >>> > -> RECONNECTED.
> > >>> >
> > >>> > On Thu, May 25, 2017 at 10:23 AM, Mark Bean  >
> > >>> wrote:
> > >>> >
> > >>> >> I reduced the number of embedded ZooKeeper servers on the 5-Node
> > NiFi
> > >>> >> Cluster from 5 to 3. This has improved the situation. I do not see
> > any
> > >>> of
> > >>> >> the three Nodes which are also ZK servers
> > disconnecting/reconnecting to
> > >>> the
> > >>> >> cluster as before. However, the two Nodes which are not running ZK
> > >>> continue
> > >>> >> to disconnect and reconnect. The following is taken from one of
> the
> > >>> non-ZK
> > >>> >> Nodes. It's curious that some messages are issued twice from the
> > same
> > >>> >> thread, but reference a different object
> > >>> >>
> > >>> >> nifi-app.log
> > >>> >> 2017-05-25 13:40:01,628 INFO [main-EventTrhead] o.a.c.f.state.
> > >>> ConnectionStateManager
> > >>> >> State change: SUSPENDED
> > >>> >> 2017-05-25 13:39:45,627 INFO [Clustering Tasks Thread-1]
> o.a.n.c.c.
> > >>> ClusterProtocolHeaertbeater
> > >>> >> Heartbeat create at 2017-05-25 13:39:45,504 and sent to FQDN:PORT
> at
> > >>> >> 2017-05-25 13:39:45,627; send took 122 millis
> > >>> >> 2017-05-25 13:39:50,862 INFO [Clustering Tasks Thread-1]
> o.a.n.c.c.
> > >>> ClusterProtocolHeaertbeater
> > >>> >> Heartbeat create at 2017-05-25 13:39:50,732 and sent to FQDN:PORT
> at
> > >>> >> 2017-05-25 13:39:50,862; send took 122 millis
> > >>> >> 2017-05-25 13:39:56,089 INFO [Clustering Tasks Thread-1]
> o.a.n.c.c.
> > >>&