Richard, Yes, this seems reasonable, but you’ll need to know how many ‘fragments’ are in each bundle. Do you have that information? If so, you can use PartitionRecord to pull that information out into attributes, and then use UpdateAttribute to make sure that the appropriate attributes are specified. Then I think you’d probably need to have a custom processor that extends BinFiles. BinFiles is an abstract class that MergeContent extends. It has a couple of different abstract methods but the important one is method:
protected abstract BinProcessingResult processBin(Bin unmodifiableBin, ProcessContext context) throws ProcessException; The others are more setup/config/validation types of thing that are likely either empty implementations or simple one-liner types of things. MergeContent would be a good example to look at to fully understand how to handle these methods. Hope this helps! -Mark On May 12, 2021, at 1:35 AM, Richard Beare <[email protected]<mailto:[email protected]>> wrote: I'm hoping to use partitionrecord as a first step, to create groups with common ID. However I cannot be certain that all the required rows will be in the same flowfile because of the way input will be chunked (e.g some row limit in the initial sql query to keep size manageable). PartitionRecord, as far as I can tell, only partitions within each flowfile - I tested by feeding the results of splitrecord into partition record. Hence I'm thinking partionrecord, update attributes using fields in each partition, mergerecord (fragmentation). I may be able to filter the complete partitions around the merge. On Wed, May 12, 2021 at 2:59 PM Chris Sampson <[email protected]<mailto:[email protected]>> wrote: PartitionRecord might be what you're after. This will allow you to analyse fields and separate records from flowfiles into chunks containing the same field values and those values will be added as flowfile attributes. https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.PartitionRecord/index.html Cheers, Chris Sampson On Tue, 11 May 2021, 23:06 Richard Beare, <[email protected]> wrote: Hi, Warning about a likely newbie question. I'm extracting records from an SQL DB that include a blob that I need to feed through some custom groovy/java. I have the basic version working using avro records throughout. However there is a complexity in that a small proportion of blobs span multiple rows and require concatenation before processing. Thus I need to ensure that all the blobs that belong together get assembled into the same flowfile before performing the concatenation (probably using a custom groovy script because there are some suffixes to remove before concatenation). I've added a "PARTS" field via my initial SQL query and there is also a sequence number column and an ID column. My plan was to use RecordPartition based on the ID, modify fragment ID and count and then reassemble with mergerecord using the defragment strategy. My problem is that I can't figure out how to get record fields into attributes. I'm hoping there is a recordpath/expression combination allowing this. Any suggestions? My fallback is to separate the initial sql queries into two parts, one for batches of single row blobs and a second that collects the multi-row ones one at a time.
