+1 on this as well. It's something I've kind of griped about before (with the loss of PostHTTP).
I don't think it would be horrible (as per Joe's concern) to offer a N:1 "bundling" property. It would just have to be stupid simple. No "groups", timeouts, correlation attributes, minimum entries, etc. It should just basically call the ProcessSession#get(int maxResults) where "maxResults" is a configurable property. Whatever number of flowfiles returned in the list is what is "bundled" into FFv3 format for output. /Adam On Fri, Sep 8, 2023 at 7:19 AM Phillip Lord <[email protected]> wrote: > +1 from me. > I’ve experimented with both methods. The simplicity of a PackageFlowfile > straight up 1:1 is convenient and straightforward. > MergeContent on the other hand can be difficult to understand and tweak > appropriately to gain desired results/throughput. > On Sep 8, 2023 at 10:14 AM -0400, Joe Witt <[email protected]>, wrote: > > Ok. Certainly simplifies it but likely makes it applicable to larger > > flowfiles only. The format is meant to allow appending and result in > large > > sets of flowfiles for io efficiency and specifically for storage as the > > small files/tons of files thing can cause poor performance pretty quickly > > (10s of thousands of files in a single directory). > > > > But maybe that simplicity is fine and we just link to the MergeContent > > packaging option if users need more. > > > > On Fri, Sep 8, 2023 at 7:06 AM Michael Moser <[email protected]> wrote: > > > > > I was thinking 1 file in -> 1 flowfile-v3 file out. No merging of > multiple > > > files at all. Probably change the mime.type attribute. It might not > even > > > have any config properties at all if we only support flowfile-v3 and > not v1 > > > or v2. > > > > > > -- Mike > > > > > > > > > On Fri, Sep 8, 2023 at 9:56 AM Joe Witt <[email protected]> wrote: > > > > > > > Mike > > > > > > > > In user terms this makes sense to me. Id only bother with v3 or > whatever > > > is > > > > latest. We want to dump the old code. And if there are seriously > older > > > > versions v1,v2 then nifi 1.x can be used. > > > > > > > > The challenge is that you end up needing some of the same complexity > in > > > > implementation and config of merge content i think. What did you > have in > > > > mind for that? > > > > > > > > Thanks > > > > > > > > On Fri, Sep 8, 2023 at 6:53 AM Michael Moser <[email protected]> > wrote: > > > > > > > > > Devs, > > > > > > > > > > I can't find if this was suggested before, so here goes. With the > > > demise > > > > > of PostHTTP in NiFi 2.0, the recommended alternative is to > > > MergeContent 1 > > > > > file into FlowFile-v3 format then InvokeHTTP. What does the > community > > > > > think about supporting a new PackageFlowFile processor that is > simple > > > to > > > > > configure (compared to MergeContent!) and simply packages flowfile > > > > > attributes + content into a FlowFile-v[1,2,3] format? This would > also > > > > > offer a simple way to export flowfiles from NiFi that could later > be > > > > > re-ingested and recovered using UnpackContent. I don't want to > submit > > > a > > > > PR > > > > > for such a processor without first asking the community whether > this > > > > would > > > > > be acceptable. > > > > > > > > > > Thanks, > > > > > -- Mike > > > > > > > > > > > > >
