I'm hoping to use partitionrecord as a first step, to create groups with common ID. However I cannot be certain that all the required rows will be in the same flowfile because of the way input will be chunked (e.g some row limit in the initial sql query to keep size manageable). PartitionRecord, as far as I can tell, only partitions within each flowfile - I tested by feeding the results of splitrecord into partition record. Hence I'm thinking partionrecord, update attributes using fields in each partition, mergerecord (fragmentation). I may be able to filter the complete partitions around the merge.
On Wed, May 12, 2021 at 2:59 PM Chris Sampson <[email protected]> wrote: > > PartitionRecord might be what you're after. This will allow you to analyse > fields and separate records from flowfiles into chunks containing the same > field values and those values will be added as flowfile attributes. > > https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.PartitionRecord/index.html > > > Cheers, > > Chris Sampson > > On Tue, 11 May 2021, 23:06 Richard Beare, <[email protected]> wrote: >> >> Hi, >> Warning about a likely newbie question. >> >> I'm extracting records from an SQL DB that include a blob that I need >> to feed through some custom groovy/java. I have the basic version >> working using avro records throughout. However there is a complexity >> in that a small proportion of blobs span multiple rows and require >> concatenation before processing. Thus I need to ensure that all the >> blobs that belong together get assembled into the same flowfile before >> performing the concatenation (probably using a custom groovy script >> because there are some suffixes to remove before concatenation). >> >> I've added a "PARTS" field via my initial SQL query and there is also >> a sequence number column and an ID column. My plan was to use >> RecordPartition based on the ID, modify fragment ID and count and then >> reassemble with mergerecord using the defragment strategy. >> >> My problem is that I can't figure out how to get record fields into >> attributes. I'm hoping there is a recordpath/expression combination >> allowing this. Any suggestions? >> >> My fallback is to separate the initial sql queries into two parts, one >> for batches of single row blobs and a second that collects the >> multi-row ones one at a time.
