Flow File Packager v3. You can find the source here:
https://github.com/apache/nifi/blob/main/nifi-commons/nifi-flowfile-packager/src/main/java/org/apache/nifi/util/FlowFilePackagerV3.java
It's a serialization format that is used for writing a flowfile (content
and attributes) to a stream (networ
Uh, sorry, "Version 3" refers to what exactly?
On 9/8/23 12:48, David Handermann wrote:
I agree that this would be a useful general feature. I also agree with
Joe that format support should be limited to*Version 3* due to the
limitations of the earlier versions.
This is definitely something th
I agree that this would be a useful general feature. I also agree with
Joe that format support should be limited to Version 3 due to the
limitations of the earlier versions.
This is definitely something that would be useful on the 1.x support
branch to provide a smooth upgrade path for NiFi 2.
Th
And also ... if we can land this in a 1.x release, this would help
tremendously to those who are going to need a replacement for PostHTTP and
don't want to "go dark" when they make the transition.
That is, without this processor in 1.x, when a user upgrades from 1.x to
2.x, they will either have t
+1 on this as well. It's something I've kind of griped about before (with
the loss of PostHTTP).
I don't think it would be horrible (as per Joe's concern) to offer a N:1
"bundling" property. It would just have to be stupid simple. No "groups",
timeouts, correlation attributes, minimum entries, etc
+1 from me.
I’ve experimented with both methods. The simplicity of a PackageFlowfile
straight up 1:1 is convenient and straightforward.
MergeContent on the other hand can be difficult to understand and tweak
appropriately to gain desired results/throughput.
On Sep 8, 2023 at 10:14 AM -0400, Joe
Ok. Certainly simplifies it but likely makes it applicable to larger
flowfiles only. The format is meant to allow appending and result in large
sets of flowfiles for io efficiency and specifically for storage as the
small files/tons of files thing can cause poor performance pretty quickly
(10s of
Most of the complexity in MergeContent is around the bundling parameters...
this processor would do no bundling, just straight pass through to the
packaging library. No worries for the user about setting max package size,
number of entries, number of bins, bin age, headers, footers, etc... even
if
I was thinking 1 file in -> 1 flowfile-v3 file out. No merging of multiple
files at all. Probably change the mime.type attribute. It might not even
have any config properties at all if we only support flowfile-v3 and not v1
or v2.
-- Mike
On Fri, Sep 8, 2023 at 9:56 AM Joe Witt wrote:
> Mik
Mike
In user terms this makes sense to me. Id only bother with v3 or whatever is
latest. We want to dump the old code. And if there are seriously older
versions v1,v2 then nifi 1.x can be used.
The challenge is that you end up needing some of the same complexity in
implementation and config of me
I have had to use that pattern myself recently. I think a simple
PackageFlowFile processor makes a lot of sense. I am +1.
Brandon
From: Michael Moser
Sent: Friday, September 8, 2023 9:52:52 AM
To: dev@nifi.apache.org
Subject: new PackageFlowFile processor
Devs,
Devs,
I can't find if this was suggested before, so here goes. With the demise
of PostHTTP in NiFi 2.0, the recommended alternative is to MergeContent 1
file into FlowFile-v3 format then InvokeHTTP. What does the community
think about supporting a new PackageFlowFile processor that is simple to
12 matches
Mail list logo
