Uh, sorry, "Version 3" refers to what exactly?
On 9/8/23 12:48, David Handermann wrote:
I agree that this would be a useful general feature. I also agree with
Joe that format support should be limited to*Version 3* due to the
limitations of the earlier versions.
This is definitely something that would be useful on the 1.x support
branch to provide a smooth upgrade path for NiFi 2.
This general topic also came up on the dev channel on the Apache NiFi
Slack group:
https://apachenifi.slack.com/archives/C0L9S92JY/p1692115270146369
One key thing to note from that discussion is supporting
interoperability with services outside of NiFi. That may be too much
of a stretch for an initial implementation, but it is something I am
planning to evaluate as time allows.
For now, something focused narrowly on FlowFile Version 3 encoding
seems like the best approach.
I recommend referencing this discussion in a new Jira issue and
outlining the general design goals.
Regards,
David Handermann
On Fri, Sep 8, 2023 at 1:11 PM Adam Taft<a...@adamtaft.com> wrote:
And also ... if we can land this in a 1.x release, this would help
tremendously to those who are going to need a replacement for PostHTTP and
don't want to "go dark" when they make the transition.
That is, without this processor in 1.x, when a user upgrades from 1.x to
2.x, they will either have to have a MergeContent/InvokeHTTP solution in
place already to replace PostHTTP, or they will have to take a (hopefully
short) outage when they bring their canvas back up (removing PostHTTP and
replacing with PackageFlowFile + InvokeHTTP).
With this processor in 1.x, they can make that transition while PostHTTP is
still available on their canvas. Wishful thinking that we can make the
entire journey from 1.x to 2.x as smooth as possible, but this could
potentially help some.
On Fri, Sep 8, 2023 at 10:55 AM Adam Taft<a...@adamtaft.com> wrote:
+1 on this as well. It's something I've kind of griped about before (with
the loss of PostHTTP).
I don't think it would be horrible (as per Joe's concern) to offer a N:1
"bundling" property. It would just have to be stupid simple. No "groups",
timeouts, correlation attributes, minimum entries, etc. It should just
basically call the ProcessSession#get(int maxResults) where "maxResults" is
a configurable property. Whatever number of flowfiles returned in the list
is what is "bundled" into FFv3 format for output.
/Adam
On Fri, Sep 8, 2023 at 7:19 AM Phillip Lord<phillord0...@gmail.com>
wrote:
+1 from me.
I’ve experimented with both methods. The simplicity of a PackageFlowfile
straight up 1:1 is convenient and straightforward.
MergeContent on the other hand can be difficult to understand and tweak
appropriately to gain desired results/throughput.
On Sep 8, 2023 at 10:14 AM -0400, Joe Witt<joe.w...@gmail.com>, wrote:
Ok. Certainly simplifies it but likely makes it applicable to larger
flowfiles only. The format is meant to allow appending and result in
large
sets of flowfiles for io efficiency and specifically for storage as the
small files/tons of files thing can cause poor performance pretty
quickly
(10s of thousands of files in a single directory).
But maybe that simplicity is fine and we just link to the MergeContent
packaging option if users need more.
On Fri, Sep 8, 2023 at 7:06 AM Michael Moser<moser...@gmail.com>
wrote:
I was thinking 1 file in -> 1 flowfile-v3 file out. No merging of
multiple
files at all. Probably change the mime.type attribute. It might not
even
have any config properties at all if we only support flowfile-v3 and
not v1
or v2.
-- Mike
On Fri, Sep 8, 2023 at 9:56 AM Joe Witt<joe.w...@gmail.com> wrote:
Mike
In user terms this makes sense to me. Id only bother with v3 or
whatever
is
latest. We want to dump the old code. And if there are seriously
older
versions v1,v2 then nifi 1.x can be used.
The challenge is that you end up needing some of the same
complexity in
implementation and config of merge content i think. What did you
have in
mind for that?
Thanks
On Fri, Sep 8, 2023 at 6:53 AM Michael Moser<moser...@gmail.com>
wrote:
Devs,
I can't find if this was suggested before, so here goes. With the
demise
of PostHTTP in NiFi 2.0, the recommended alternative is to
MergeContent 1
file into FlowFile-v3 format then InvokeHTTP. What does the
community
think about supporting a new PackageFlowFile processor that is
simple
to
configure (compared to MergeContent!) and simply packages flowfile
attributes + content into a FlowFile-v[1,2,3] format? This would
also
offer a simple way to export flowfiles from NiFi that could later
be
re-ingested and recovered using UnpackContent. I don't want to
submit
a
PR
for such a processor without first asking the community whether
this
would
be acceptable.
Thanks,
-- Mike
