I have a field in the record saying how many parts the blob has been broken into - how do I put that information into an attribute?
On Thu, May 13, 2021 at 5:36 AM Mark Payne <[email protected]> wrote: > > Richard, > > Yes, this seems reasonable, but you’ll need to know how many ‘fragments’ are > in each bundle. Do you have that information? If so, you can use > PartitionRecord to pull that information out into attributes, and then use > UpdateAttribute to make sure that the appropriate attributes are specified. > Then I think you’d probably need to have a custom processor that extends > BinFiles. BinFiles is an abstract class that MergeContent extends. It has a > couple of different abstract methods but the important one is method: > > protected abstract BinProcessingResult processBin(Bin unmodifiableBin, > ProcessContext context) throws ProcessException; > > The others are more setup/config/validation types of thing that are likely > either empty implementations or simple one-liner types of things. > MergeContent would be a good example to look at to fully understand how to > handle these methods. > > Hope this helps! > -Mark > > > On May 12, 2021, at 1:35 AM, Richard Beare <[email protected]> wrote: > > I'm hoping to use partitionrecord as a first step, to create groups > with common ID. However I cannot be certain that all the required rows > will be in the same flowfile because of the way input will be chunked > (e.g some row limit in the initial sql query to keep size manageable). > PartitionRecord, as far as I can tell, only partitions within each > flowfile - I tested by feeding the results of splitrecord into > partition record. Hence I'm thinking partionrecord, update attributes > using fields in each partition, mergerecord (fragmentation). I may be > able to filter the complete partitions around the merge. > > On Wed, May 12, 2021 at 2:59 PM Chris Sampson <[email protected]> > wrote: > > > PartitionRecord might be what you're after. This will allow you to analyse > fields and separate records from flowfiles into chunks containing the same > field values and those values will be added as flowfile attributes. > > https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.PartitionRecord/index.html > > > Cheers, > > Chris Sampson > > On Tue, 11 May 2021, 23:06 Richard Beare, <[email protected]> wrote: > > > Hi, > Warning about a likely newbie question. > > I'm extracting records from an SQL DB that include a blob that I need > to feed through some custom groovy/java. I have the basic version > working using avro records throughout. However there is a complexity > in that a small proportion of blobs span multiple rows and require > concatenation before processing. Thus I need to ensure that all the > blobs that belong together get assembled into the same flowfile before > performing the concatenation (probably using a custom groovy script > because there are some suffixes to remove before concatenation). > > I've added a "PARTS" field via my initial SQL query and there is also > a sequence number column and an ID column. My plan was to use > RecordPartition based on the ID, modify fragment ID and count and then > reassemble with mergerecord using the defragment strategy. > > My problem is that I can't figure out how to get record fields into > attributes. I'm hoping there is a recordpath/expression combination > allowing this. Any suggestions? > > My fallback is to separate the initial sql queries into two parts, one > for batches of single row blobs and a second that collects the > multi-row ones one at a time. > >
