Hey Jeremy, Something linke this https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.DetectDuplicate/index.html <https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.DetectDuplicate/index.html> ?
> On 15. Feb 2021, at 04:45, Jeremy Pemberton-Pigott <fuzzych...@gmail.com> > wrote: > > Hi everyone, I'm wondering if there is a Detect Duplicate processor that can > read records from a flow file and as output gives just the non-duplicates > (can be single records or a group of non-duplicates would be better). I want > to use a record reader to avoid splitting the json content into 10000s of > flow files to detect the duplicates. Immediately after this flow is a record > reader/writer going to HBase. > > Jeremy
