Apache MiNiFi C++ 0.3.0 Release Helper Guide

[Marc]

Hello Apache NiFi community,

Please find the associated guidance to help those interested in
validating/verifying the release so they can vote.

# Download latest KEYS file:
https://dist.apache.org/repos/dist/dev/nifi/KEYS

# Import keys file:
gpg --import KEYS

# Pull down nifi-minifi-cpp-0.3.0 source release artifacts for review:

wget https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/
0.3.0/nifi-minifi-cpp-0.3.0-source.tar.gz
wget https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/
0.3.0/nifi-minifi-cpp-0.3.0-source.tar.gz.asc
wget https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/
0.3.0/nifi-minifi-cpp-0.3.0-source.tar.gz.md5
wget https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/
0.3.0/nifi-minifi-cpp-0.3.0-source.tar.gz.sha1
wget https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/
0.3.0/nifi-minifi-cpp-0.3.0-source.tar.gz.sha256

# Verify the signature
gpg --verify nifi-minifi-cpp-0.3.0-source.tar.gz.asc

# Verify the hashes (md5, sha1, sha256) match the source and what was
provided in the vote email thread
md5sum nifi-minifi-cpp-0.3.0-source.tar.gz
sha1sum nifi-minifi-cpp-0.3.0-source.tar.gz
sha256sum nifi-minifi-cpp-0.3.0-source.tar.gz

# Untar nifi-minifi-cpp-0.3.0-source.tar.gz
tar xf nifi-minifi-cpp-0.3.0-source.tar.gz

# Verify the build works

Be mindful of the pre-requisites required for the C++ version of MiNiFi,
enumerated in the README [1] and the switching to the CMake build system
These can vary from system to system and distribution, an example of the
package listing for a recent Ubuntu release is:
  cmake gcc g++ libcurl4-openssl-dev uuid-dev uuid libboost-all-dev
libssl-dev doxygen libpython3-dev

Once the required environment is established, a build with testing and
linting can be performed via:

  cd nifi-minifi-cpp-0.3.0-source
  mkdir build
  cd build
  cmake ..
  make package
  make test
  make linter

# Verify the contents contain a good README, NOTICE, and LICENSE.

# Verify the git commit ID is correct

# Verify the RC was branched off the correct git commit ID

# Look at the resulting convenience assembly (nifi-minifi-cpp-0.3.0-bin.tar.gz)
found in your build directory

# Make sure the README, NOTICE, and LICENSE are present and correct

# Run the resulting convenience binary and make sure it works as expected

Be mindful of caveats for this initial release, listed in the README. Since
the convenience binaries include
the scripting extension [2], Python support is needed per our README [1]

For some additional assistance, a package with configuration files for both
a MiNiFI instance and a NiFi instance available at https://cwiki.apache.org/
confluence/display/MINIFI/Releasing+MiNiFi#ReleasingMiNiFi-
SampleNiFiandMiNiFiConfigurationtotransmitdatafromMiNiFitoNiFiviaSitetoSite
The provided sample configuration bundle assumes that NiFi is configured to
listen on port 8081 and has 10001 configured for Site to Site's
nifi.remote.input.socket.port.

# Send a response to the vote thread indicating a +1, 0, -1 based on your
findings.

Thank you for your time and effort to validate the release! Please let me
know if you have any questions or need clarification.

Thanks,
Marc

[1] https://github.com/apache/nifi-minifi-cpp/blob/MINIFICPP-304-RC0-0.3.0/
README.md#system-requirements
[2] https://cwiki.apache.org/confluence/pages/viewpage.
action?pageId=74685143

[VOTE] Release Apache NiFi MiNiFI C++ 0.3.0 (RC1)

[Marc]

Hello Apache NiFi Community,

I am pleased to be calling this vote for the source release of Apache NiFi
MiNiFi C++, nifi-minifi-cpp-0.3.0.

The source archive, signature, and digests can be located at:

Source Archive:

https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/0.3.0/nifi-minifi-cpp-0.3.0-source.tar.gz
GPG armored signature:

https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/0.3.0/nifi-minifi-cpp-0.3.0-source.tar.gz.asc
Source MD5:

https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/0.3.0/nifi-minifi-cpp-0.3.0-source.tar.gz.md5
Source SHA1:

https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/0.3.0/nifi-minifi-cpp-0.3.0-source.tar.gz.sha1
Source SHA256:

https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/0.3.0/nifi-minifi-cpp-0.3.0-source.tar.gz.sha256

The Git tag is minifi-cpp-0.3.0-RC1
The Git commit hash is d3852a73beaafa78a789d975f6d3a595b7761d41
*
https://git-wip-us.apache.org/repos/asf?p=nifi-minifi-cpp.git;a=commit;h=d3852a73beaafa78a789d975f6d3a595b7761d41
*
https://github.com/apache/nifi-minifi-cpp/commit/d3852a73beaafa78a789d975f6d3a595b7761d41

Checksums of nifi-minifi-cpp-0.3.0-source.tar.gz:
MD5: 8bcc8987e8322e1be0ddc631adc4f9bd
SHA1: 811cc8c54572f25121b64b123a8fca176e81509b
SHA256: f87815c31b5b15a30d2261800a89d9eab678d5ecc485d53f83c035d6ddb31b8a

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/phrocker

KEYS file available here:
https://dist.apache.org/repos/dist/dev/nifi/KEYS

59 issues were closed/resolved for this release:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12321520=12341640

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/MINIFI/Release+Notes#ReleaseNotes-Versioncpp-0.3.0

Since Thursday is a major US Holiday, the vote will be open for 96 hours
and will close on 25 Nov at 3PM EDT [1].

Please download the release candidate and evaluate the necessary items
including checking hashes, signatures, build from source, and test. Then
please vote:

[ ] +1 Release this package as nifi-minifi-cpp-0.3.0
[ ] +0 no opinion
[ ] -1 Do not release this package because...

Thanks!


[1] You can determine this time for your local time zone at
https://s.apache.org/minifi-cpp-0.3.0-rc1-close

Re: Provenance query results in Node disconnect from cluster

[Mark Bean]

Thanks for some clarifying information. Setting
nifi.cluster.node.read.timeout=30
sec seems to have alleviated the problem.

It was determined that there was a relatively long time in performing all
the authorizations or each Provenance Event after choosing Global Menu ->
Data Provenance. In this case, the Provenance Query Thread authorizes "Data
for ..." for each processor. Each such authorization takes approximately
0.5-0.6 ms. (Timing was taken with custom authorization logic disabled.) I
have not yet determined if this authorization proceeds for ALL Provenance
Events, or only for the 1,000 events which the UI limits for display
purposes. I have noted that all authorizations are being handled by a
single Provenance Query Thread despite the property
nfi.provenance.repository.query.threads=2.
I assume this property allows more threads for simultaneous client
requests, but each individual request uses only a single thread.

Also, it was determined that GC was not a significant factor. The JVM is
spending approximately 10% of its time performing GC, but none of it a full
GC. And, the time of any one GC is reasonable (approx. 0.5 sec).


On Mon, Nov 20, 2017 at 4:11 PM, Mark Payne  wrote:

> Mark,
>
> By and large, when you run into issues with timeouts on cluster
> replication, in my experience, the culprit
> is usually Garbage Collection. So it may be that you are not
> thread-limited or CPU-limited,
> or resource limited at all, just that garbage collection is kicking in at
> an inopportune time. In such a situation,
> my suggestion would be to use a nifi.cluster.node.read.timeout of say 30
> seconds instead of 10, and to
> look into how the garbage collection is performing on your system.
>
> I have answered specific questions below, though, in case they are helpful.
>
> Thanks
> -Mark
>
>
> > On Nov 20, 2017, at 3:25 PM, Mark Bean  wrote:
> >
> > We are seeing cases where a user attempts to query provenance on a
> cluster.
> > One or more Nodes may not respond to the request in a timely manner, and
> is
> > then subsequently disconnected from the cluster. The nifi-app.log shows
> log
> > messages similar to:
> >
> > ThreadPoolRequestReplicator Failed to replicate request POST
> > /nifi-api/provenance to {host:port} due to
> > com.sun.jersy.api.client.ClientHandlerException:
> > java.net.SocketTimeoutException: Read timed out
> > NodeClusterCoordinator The following nodes failed to process URI
> > /nifi-api/provenance '{list of one or more nodes}'. Requesting each node
> > disconnect from cluster.
> >
> > We have implemented a custom authorizer. For certain policies, additional
> > authorization checking is performed. Provenance is one such policy which
> > performs additional checking. It is surprising that the process is taking
> > so long as to time out the request. Currently, timeouts are set as:
> > nifi.cluster.node.read.timeout=10 sec
> > nifi.cluster.request.replication.claim.timeout=30 sec
> >
> > This leads me to believe we are thread-limited, not CPU-limited.
> >
> > In this scenario, what threads are involved? Would
> > nifi.cluster.node.protocol.threads (or .max.threads) be limiting the
> > processing of such api calls?
>
> >>> These are the jetty threads that are involved, on the 'receiving' side
> and the nifi.cluster.node.protocol.threads on the client side
>
> >
> > Is the api provenance request(s) limited by
> > nifi.provenance.repository.query.thread?
>
> >>> These query threads are background threads that are used to populate
> the results
> of the query. Client requests will not block on those results.
>
> >
> > Are there other thread-related properties we should be looking at?
> >
>
> >>> I don't think so. I can't think of any off of the top of my head,
> anyway.
>
> > Are thread properties (such as nifi.provenance.repository.query.threads)
> > counted against the total threads given by nifi.web.jetty.threads?
>
> >>> No, these are separate thread pools.
>
> >
> > Thanks,
> > Mark
>
>

Re: [ANNOUNCE] New NiFi PMC member Mike Moser

[Andy Christianson]

Congrats!

Sent from [ProtonMail](https://protonmail.com), Swiss-based encrypted email.

>  Original Message 
> Subject: [ANNOUNCE] New NiFi PMC member Mike Moser
> Local Time: November 20, 2017 4:18 PM
> UTC Time: November 20, 2017 9:18 PM
> From: tk...@apache.org
> To: dev@nifi.apache.org
>
> NiFi community,
> On behalf of the Apache NiFi PMC, I am pleased to announce that Mike Moser
> has accepted the PMC's invitation to join the Apache NiFi PMC. We greatly
> appreciate all of Mike's hard work and generous contributions to the
> project. We look forward to continued involvement in the project.
>
> Mike has been a long time code contributor to NiFi, he performed the final
> release management duties for the 0.x line, has been involved with
> verifying releases and always a delight on the mailing lists.
>
> Please join us in congratulating and welcoming Mike to the Apache NiFi PMC.
>
> Congratulations and welcome, Mike!
>
> -- Tony

Re: [ANNOUNCE] New NiFi PMC member Mike Moser

[Joe Skora]

Congrats Mike and welcome aboard!

On Tue, Nov 21, 2017 at 7:34 AM, Mark Bean  wrote:

> Congrats Mike!!
>
>
> On Mon, Nov 20, 2017 at 7:17 PM, Corey Flowers 
> wrote:
>
> > About time!!! Congrats Mike
> >
> > Sent from my iPhone
> >
> > > On Nov 20, 2017, at 6:58 PM, Michael Hogue <
> michael.p.hogu...@gmail.com>
> > wrote:
> > >
> > > Congratulations, Mike!
> > >> On Mon, Nov 20, 2017 at 17:59 Jeff  wrote:
> > >>
> > >> Congrats Mike!
> > >>
> > >>> On Mon, Nov 20, 2017 at 4:46 PM Matt Burgess 
> > wrote:
> > >>>
> > >>> Congrats and welcome aboard!
> > >>>
> > >>> On Mon, Nov 20, 2017 at 4:40 PM, Scott Aslan 
> > >>> wrote:
> >  Congrats!
> > 
> >  On Mon, Nov 20, 2017 at 4:25 PM, Juan Sequeiros <
> helloj...@gmail.com>
> > >>> wrote:
> > 
> > > Mike this is awesome. Congrats and thanks.
> > >
> > >> On Mon, Nov 20, 2017 at 4:20 PM Joe Witt 
> > wrote:
> > >>
> > >> Congrats Mike!  Thanks for all your efforts
> > >>
> > >> On Mon, Nov 20, 2017 at 4:18 PM, Tony Kurc 
> > >> wrote:
> > >>> NiFi community,
> > >>> On behalf of the Apache NiFi PMC, I am pleased to announce that
> > >> Mike
> > >> Moser
> > >>> has accepted the PMC's invitation to join the Apache NiFi PMC.
> We
> > >> greatly
> > >>> appreciate all of Mike's hard work and generous contributions to
> > >> the
> > >>> project. We look forward to continued involvement in the project.
> > >>>
> > >>> Mike has been a long time code contributor to NiFi, he performed
> > >> the
> > >> final
> > >>> release management duties for the 0.x line, has been involved
> with
> > >>> verifying releases and always a delight on the mailing lists.
> > >>>
> > >>> Please join us in congratulating and welcoming Mike to the Apache
> > >>> NiFi
> > >> PMC.
> > >>>
> > >>> Congratulations and welcome, Mike!
> > >>>
> > >>> -- Tony
> > >>
> > >
> > >>>
> > >>
> >
>

Re: [ANNOUNCE] New NiFi PMC member Mike Moser

[Mark Bean]

Congrats Mike!!


On Mon, Nov 20, 2017 at 7:17 PM, Corey Flowers 
wrote:

> About time!!! Congrats Mike
>
> Sent from my iPhone
>
> > On Nov 20, 2017, at 6:58 PM, Michael Hogue 
> wrote:
> >
> > Congratulations, Mike!
> >> On Mon, Nov 20, 2017 at 17:59 Jeff  wrote:
> >>
> >> Congrats Mike!
> >>
> >>> On Mon, Nov 20, 2017 at 4:46 PM Matt Burgess 
> wrote:
> >>>
> >>> Congrats and welcome aboard!
> >>>
> >>> On Mon, Nov 20, 2017 at 4:40 PM, Scott Aslan 
> >>> wrote:
>  Congrats!
> 
>  On Mon, Nov 20, 2017 at 4:25 PM, Juan Sequeiros 
> >>> wrote:
> 
> > Mike this is awesome. Congrats and thanks.
> >
> >> On Mon, Nov 20, 2017 at 4:20 PM Joe Witt 
> wrote:
> >>
> >> Congrats Mike!  Thanks for all your efforts
> >>
> >> On Mon, Nov 20, 2017 at 4:18 PM, Tony Kurc 
> >> wrote:
> >>> NiFi community,
> >>> On behalf of the Apache NiFi PMC, I am pleased to announce that
> >> Mike
> >> Moser
> >>> has accepted the PMC's invitation to join the Apache NiFi PMC.  We
> >> greatly
> >>> appreciate all of Mike's hard work and generous contributions to
> >> the
> >>> project. We look forward to continued involvement in the project.
> >>>
> >>> Mike has been a long time code contributor to NiFi, he performed
> >> the
> >> final
> >>> release management duties for the 0.x line, has been involved with
> >>> verifying releases and always a delight on the mailing lists.
> >>>
> >>> Please join us in congratulating and welcoming Mike to the Apache
> >>> NiFi
> >> PMC.
> >>>
> >>> Congratulations and welcome, Mike!
> >>>
> >>> -- Tony
> >>
> >
> >>>
> >>
>

答复: Re: How to delete the data in the flowfile?

[luby]

Very appreicate for your helpl. It's very helpful. :)



发件人: 
Jeff 
收件人:
dev@nifi.apache.org
日期:
2017/11/20 22:07
主题:
Re: How to delete the data in the flowfile?



Hello Boying,

Once flowfiles have completed processing, they may still be archived 
within
the content repository for a certain period of time before they age-off. 
In
the NiFi Admin guide, there is a section on Content Repository properties
[1] you can set in nifi.properties, through which you can tweak how much
space is used to archive, how long flowfiles are archived, or to disable
archiving completely.

Lowering the "nifi.content.repository.archive.max.retention.period" and
"nifi.content.repository.archive.max.usage.percentage" properties can help
limit the amount of disk space the content repository uses for archived
flowfiles.  You can disable content archiving by setting
"nifi.content.repository.archive.enabled" to false if you prefer to have 
no
archive at all.

If your flow uses a processor like PutFile to place a flowfile in a
temporary directory to do further processing on it, or to allow "backups"
of the flowfile for various stages of processing, then your flow must be
designed to clean up those files after they are no longer needed.  There
are several ways to do this, one of them being Wait/Notify processors.
There's a blog that Koji has written [2] with some examples on how to use
the Wait and Notify processors, and the concepts covered in the blog 
should
be usable in your case where you might want to use the Wait/Notify
processors to signal that flowfiles that are no longer needed that have
been explicitly archived/copied by processors like "PutFile" can be 
removed.

Please let me know if neither of these solutions help with disk space
issues while using your flow.  If you provide your flow as an example, we
can take a look at other ways to try to minimize disk usage.

[1]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#file-system-content-repository-properties

[2]
http://ijokarumawak.github.io/nifi/2017/02/02/nifi-notify-batch/#alternative-solution-waitnotify


On Mon, Nov 20, 2017 at 3:16 AM  wrote:

> Hi, All,
>
> We use NiFi to import data from Oracle database to Hive.
>
> The first step is to extract all data from the Oracle database and 
persist
> it into the flowfile
> which will then 'flow' into other processors to do further processing.
>
> After persisting the data into the Hive, we found that the data 
persisted
> in the first step were not
> deteled. This will occupied a lot of disk spaces.
>
> So is there any way to tell NiFi to delete those data after the next
> processor has finished reading the data?
>
> Thanks
>
> Boying
>
>
>
>
> 本邮件内容包含保密信息。如阁下并非拟发送的收件人，请您不要阅读、保存、对
外
> 披露或复制本邮件的任何内容，或者打开本邮件的任何附件。请即回复邮件告知发
件
> 人，并立刻将该邮件及其附件从您的电脑系统中全部删除，不胜感激。
>
>
> This email message may contain confidential and/or privileged 
information.
> If you are not the intended recipient, please do not read, save, 
forward,
> disclose or copy the contents of this email or open any file attached to
> this email. We will be grateful if you could advise the sender 
immediately
> by replying this email, and delete this email and any attachment or 
links
> to this email completely and immediately from your computer system.
>
>
>
>





 
本邮件内容包含保密信息。如阁下并非拟发送的收件人，请您不要阅读、保存、对外
披露或复制本邮件的任何内容，或者打开本邮件的任何附件。请即回复邮件告知发件
人，并立刻将该邮件及其附件从您的电脑系统中全部删除，不胜感激。

 
This email message may contain confidential and/or privileged information. 
If you are not the intended recipient, please do not read, save, forward, 
disclose or copy the contents of this email or open any file attached to 
this email. We will be grateful if you could advise the sender immediately 
by replying this email, and delete this email and any attachment or links 
to this email completely and immediately from your computer system. 

Re: Current state of Kafka processors

[Woschitz, Janosch]

Hi Joe,

Thanks a lot for the explanation, this was also my assumption by looking at the 
code, though I assumed I missed something.

I filed https://issues.apache.org/jira/browse/NIFI-4623 and added a simple PR 
which just removes the warning in the doc comments for the newer Kafka 
processors.

Regards,
Janosch




On 21.11.17, 00:24, "Joe Witt"  wrote:

>Janosch
>
>Sorry for the lack of response.  It was there previously because it
>was observed that timeout conditions could cause threads to stay in a
>hung state and as the caller of the client we weren't in a position to
>resolve it.  However, newer client libraries are better and I no
>longer have observed these issues personally.  For the 0_10 or newer
>clients I think we can adjust the docs to indicate that these should
>work just fine.  Please feel free to file a JIRA if you dont mind.
>
>Thanks
>
>On Mon, Nov 20, 2017 at 6:22 PM, Woschitz, Janosch
> wrote:
>> Hi everyone,
>>
>> Has anyone an idea what's the technical background regarding the warning in 
>> the Kafka processors documentation?
>>
>> It is not clear to me how the processor can end up in a "indefinite stuck 
>> state”.
>>
>> Thanks,
>> Janosch
>>
>>
>>
>> On 14.11.17, 11:01, "Woschitz, Janosch" 
>>  wrote:
>>
>>>Hi,
>>>
>>>I saw the following notice on the current documentation for the PublishKafka 
>>>and ConsumeKafka processors:
>>>
>>>"Please note there are cases where the publisher can get into an indefinite 
>>>stuck state. We are closely monitoring how this evolves in the Kafka 
>>>community and will take advantage of those fixes as soon as we can. In the 
>>>mean time it is possible to enter states where the only resolution will be 
>>>to restart the JVM NiFi runs on.”
>>>
>>>I was unable to find related and unresolved JIRA issues and was wondering if 
>>>this is still an issue in NiFi 1.4.0?
>>>
>>>Could somebody provide more insights into the current state?
>>>
>>>Thanks,
>>>Janosch
>>>