A working solution: PartitionRecord on blobID and blobSize, route on attribute for blobSize = record.count. The non matching branch goes through PartitionRecord again, this time on blobID, blobSize and blobIndex, which functions as split, but attaching content as attributes. Then through updateattributes to copy those attributes to the appropriate "fragment" attribute, then to merge.
On Thu, May 13, 2021 at 5:36 AM Mark Payne <[email protected]> wrote: > > Richard, > > Yes, this seems reasonable, but you’ll need to know how many ‘fragments’ are > in each bundle. Do you have that information? If so, you can use > PartitionRecord to pull that information out into attributes, and then use > UpdateAttribute to make sure that the appropriate attributes are specified. > Then I think you’d probably need to have a custom processor that extends > BinFiles. BinFiles is an abstract class that MergeContent extends. It has a > couple of different abstract methods but the important one is method: > > protected abstract BinProcessingResult processBin(Bin unmodifiableBin, > ProcessContext context) throws ProcessException; > > The others are more setup/config/validation types of thing that are likely > either empty implementations or simple one-liner types of things. > MergeContent would be a good example to look at to fully understand how to > handle these methods. > > Hope this helps! > -Mark > > > On May 12, 2021, at 1:35 AM, Richard Beare <[email protected]> wrote: > > I'm hoping to use partitionrecord as a first step, to create groups > with common ID. However I cannot be certain that all the required rows > will be in the same flowfile because of the way input will be chunked > (e.g some row limit in the initial sql query to keep size manageable). > PartitionRecord, as far as I can tell, only partitions within each > flowfile - I tested by feeding the results of splitrecord into > partition record. Hence I'm thinking partionrecord, update attributes > using fields in each partition, mergerecord (fragmentation). I may be > able to filter the complete partitions around the merge. > > On Wed, May 12, 2021 at 2:59 PM Chris Sampson <[email protected]> > wrote: > > > PartitionRecord might be what you're after. This will allow you to analyse > fields and separate records from flowfiles into chunks containing the same > field values and those values will be added as flowfile attributes. > > https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.PartitionRecord/index.html > > > Cheers, > > Chris Sampson > > On Tue, 11 May 2021, 23:06 Richard Beare, <[email protected]> wrote: > > > Hi, > Warning about a likely newbie question. > > I'm extracting records from an SQL DB that include a blob that I need > to feed through some custom groovy/java. I have the basic version > working using avro records throughout. However there is a complexity > in that a small proportion of blobs span multiple rows and require > concatenation before processing. Thus I need to ensure that all the > blobs that belong together get assembled into the same flowfile before > performing the concatenation (probably using a custom groovy script > because there are some suffixes to remove before concatenation). > > I've added a "PARTS" field via my initial SQL query and there is also > a sequence number column and an ID column. My plan was to use > RecordPartition based on the ID, modify fragment ID and count and then > reassemble with mergerecord using the defragment strategy. > > My problem is that I can't figure out how to get record fields into > attributes. I'm hoping there is a recordpath/expression combination > allowing this. Any suggestions? > > My fallback is to separate the initial sql queries into two parts, one > for batches of single row blobs and a second that collects the > multi-row ones one at a time. > >
