Fork-Join Enrichment, Input Dependent Behaviour

[stephen.hindmarch]

HI,

I seem to have a situation similar to 
[NIFI-9903], this time in the 
JoinEnrichment processor.

I am performing some enrichments, and sometimes the enrichment look up fails as 
the item is not there. If the enrichment fails for the first item in an array 
of records then the merge fails to properly merge subsequent records in the 
same array. The join processor uses an "infer" JSON reader for the original 
leg, a schema based reader for the enriched leg, and a "inherit" JSON writer. I 
am using the "Insert" join strategy. Here is an example.

If the original record is

[{"transport_protocol_id":17,"Enrichment":{}},{"transport_protocol_id":6,"Enrichment":{}}]

Then both lookups succeed, and the enrichment record looks like this

[
  {"network_transport" : {"name" : "udp", "code" : 17, "alias" : "UDP", 
"comment" : "user datagram protocol"}},
  {"network_transport" : {"name" : "tcp", "code" : 6, "alias" : "TCP", 
"comment" : "transmission control protocol"}
]

And the joined record looks like this.

[
  {"transport_protocol_id" : 17,"Enrichment" : {
"network_transport" : {"name" : "udp","code" : 17,"alias" : "UDP","comment" 
: "user datagram protocol"}}},
  {"transport_protocol_id" : 6,"Enrichment" : {
"network_transport" : {"name" : "tcp","code" : 6,"alias" : "TCP","comment" 
: "transmission control protocol"}}}
]

However if the first record has a key value that is out of range, such as this

[{"transport_protocol_id":,"Enrichment":{}},{"transport_protocol_id":6,"Enrichment":{}}]

Then the first record in the enriched leg will be null, even if the rest of the 
records are correct. However the enrichment is still valid JSON once I have 
processed it in the enrichment leg.

[
  {"network_transport" : null},
  {"network_transport" : {"name" : "tcp", "code" : 6,"alias" : "TCP", "comment" 
: "transmission control protocol"}}
]

But the joined record does not properly process the subsequent records, and the 
content looks like this.

[
  {"transport_protocol_id" : ,"Enrichment" : {
"network_transport" : null}},
  {"transport_protocol_id" : 6,"Enrichment" : {
"network_transport" : "MapRecord[{name=tcp, alias=TCP, comment=transmission 
control protocol, code=6}]"}}
]

Is there any step I could use to ensure the join happens as expected? Or is 
this the same situation as the JIRA I mentioned above? I am not able to use a 
schema based writer as our real case has too many input record types and 
enrichment options that the number of combinations, and hence schemas, could 
not be managed.

Thanks

Steve Hindmarch

RE: Multiple GitLab accounts? (Nifi Registry)

[stephen.hindmarch]

You can have multiple registries, so you could put the buckets for the second 
account in a different registry, if that suits your use case.

Steve Hindmarch

From: Jakobsson Stefan 
Sent: 16 June 2022 11:27
To: users@nifi.apache.org
Subject: RE: Multiple GitLab accounts? (Nifi Registry)

The is exactly the solution we wanted to use, but I guess we need to use some 
other solution for now.

Will create a Jira ticket.

Stefan Jakobsson

Systems Manager  |  Scania IT, IACA |  Scania CV AB
Phone: +46 8 553 527 27 Mobile: +46 7 008 834 76
Forskargatan 20, SE-151 87 Södertälje, Sweden
stefan.jakobs...@scania.com

From: Pierre Villard 
mailto:pierre.villard...@gmail.com>>
Sent: den 16 juni 2022 11:49
To: users@nifi.apache.org
Subject: Re: Multiple GitLab accounts? (Nifi Registry)

No, it's not an option at this time. I imagine you'd like to have the option to 
associate a given bucket to a given repository?
It'd require quite some changes also involving some UI/UX improvements. You can 
file a JIRA so we can keep track of this.

Thanks,
Pierre

Le jeu. 16 juin 2022 à 11:42, Jakobsson Stefan 
mailto:stefan.jakobs...@scania.com>> a écrit :
Hi Pierre

Connecting one Registry to multiple Gitlab repositories or at least 2 different 
ones for now, possibly more in the future

Stefan Jakobsson

Systems Manager  |  Scania IT, IACA |  Scania CV AB
Phone: +46 8 553 527 27 Mobile: +46 7 008 834 76
Forskargatan 20, SE-151 87 Södertälje, Sweden
stefan.jakobs...@scania.com

From: Pierre Villard 
mailto:pierre.villard...@gmail.com>>
Sent: den 16 juni 2022 11:37
To: users@nifi.apache.org
Subject: Re: Multiple GitLab accounts? (Nifi Registry)

Hi Stefan,

Are you looking at connecting one NiFi Registry to multiple Gitlab 
repositories? or have multiple users used by the NiFi Registry to perform the 
git operations against one repository?

Pierre

Le jeu. 16 juin 2022 à 08:11, Jakobsson Stefan 
mailto:stefan.jakobs...@scania.com>> a écrit :
Any info on this query?

Stefan Jakobsson

Systems Manager  |  Scania IT, IACA |  Scania CV AB
Phone: +46 8 553 527 27 Mobile: +46 7 008 834 76
Forskargatan 20, SE-151 87 Södertälje, Sweden
stefan.jakobs...@scania.com

From: Jakobsson Stefan 
mailto:stefan.jakobs...@scania.com>>
Sent: den 14 juni 2022 10:46
To: users@nifi.apache.org
Subject: Multiple GitLab accounts? (Nifi Registry)

Hi guys!

Is it possible to configure nifi registry to use multiple gitlab accounts?

And if so, how?

Stefan Jakobsson

Systems Manager  |  Scania IT, IACA |  Scania CV AB
Phone: +46 8 553 527 27 Mobile: +46 7 008 834 76
Forskargatan 20, SE-151 87 Södertälje, Sweden
stefan.jakobs...@scania.com

RE: batch job state tracking

[stephen.hindmarch]

The traditional way is to maintain some kind of metadata or log table in the 
same database you are reading from or writing to, whichever makes sense. With 
NiFi you can use PutSQL and GetSQL as Isha describes, or make use of the 
various cache services.

Steve Hindmarch

From: Isha Lamboo 
Sent: 16 June 2022 08:58
To: users@nifi.apache.org
Subject: RE: batch job state tracking

For manual checks, NiFi provides the provenance data. You can check the last 
processor in the flow to see recent flowfiles and their lineage.
For a more systematic approach you can use any number of external systems to 
help you track this, depending on what’s available.

The simplest is to use LogMessage processors and optionally configure Logback 
to send those to a different logfile. You’re most likely already using a log 
analyzer stack, so a query there should give you the insight you need.

Another way I have good experience with is to use a database table. Insert a 
record when the process starts (using PutSQL) and then update it with either 
success or failure and timestamps. That has the added benefit of being able to 
retrieve the last run’s status from the NiFi flow.

Regards,

Isha

Van: K Makeev mailto:k.mak...@gmail.com>>
Verzonden: woensdag 15 juni 2022 19:05
Aan: users@nifi.apache.org
Onderwerp: batch job state tracking

Hi

Let's say I have a flow which reads the whole database table and puts all its 
content to another database table.
It runs in a batch mode - once a day.

How can I track its state - like when it was completed?

I know that Nifi is a stream thing but may there some new features to deal with 
it

Thank you

Kirill

RE: Unexpected Behaviour In LookupRecord With "Route To success" Strategy

[stephen.hindmarch]

Thanks Mark,

Is there a JIRA open for this?

Regards
Steve


From: Mark Payne 
Sent: 11 April 2022 14:34
To: users@nifi.apache.org
Subject: Re: Unexpected Behaviour In LookupRecord With "Route To success" 
Strategy

Steve,

Thanks for the note. Ironically, I ran into this issue just yesterday. 
Unfortunately, it’s a bug that will have to be addressed.

In the meantime, if you define the schema for your Record Writer explicitly, it 
should work as expected. The issue comes down to the fact that the first record 
is enriched. And then the schema is determined from the enriched data. Then the 
rest are enriched. But if the first one doesn’t have any enrichment data added, 
the result is that the schema is determined for the flowfile without any 
enrichment. So while the records do get enriched, the schema that is associated 
with the FlowFile is missing those fields. So explicitly defining the schema 
should work.

Thanks
-Mark



On Apr 11, 2022, at 9:08 AM, 
stephen.hindma...@bt.com wrote:

Hi all,

I am trying to set up a simple enrichment pipeline where flow records get 
enriched from a Redis Distributed Map cache and I use a sequence of 
LookupRecord processors to gather the enrichment data. I am using the “Route to 
success” routing strategy because I would like to avoid fragmenting my flow 
files. However, the results are not what I expected and if the first record 
does not match an enrichment record then no records get enriched.

Here is a simple test case I have created.

1: Create a lookup record processor with the following parameters
Result RecordPath = /mood
Routing Strategy = Route To Success
key = concat('mood/',name)

2: Add these keys to my Redis index.
set mood/fred happy
set mood/charlie sad

3: Send in this flow file
[{"name":"fred"},{"name":"bill"},{"name":"charlie"}]

4: View the result
[{"name":"fred","mood":"happy"},{"name":"bill","mood":null},{"name":"charlie","mood":"sad"}]

That looks OK, every lookup has happened, and I can see that Bill was not 
matched as the enriched value is null.

5: Now try a different flow file, with Bill first.
[{"name":"bill"},{"name":"fred"},{"name":"charlie"}]

Result
[{"name":"bill"},{"name":"fred"},{"name":"charlie"}]

So because the first record did not match, no matches are made, and it looks as 
if the processing never happened.

6: Change the routing strategy to “Route to matched/unmatched”. The result is
Matched => [{"name":"fred","mood":"happy"},{"name":"charlie","mood":"sad"}]
Unmatched => [{"name":"bill"}]

So I have achieved all of my lookups, but the cost is I have fragmented my flow 
file. After 4 lookups my original flow file (which in production will have a. 
1000 records) will have been fragmented into 16 separate files, with a 
consequent impact on performance. Also the indication that the unmatched record 
was not matched is lost, which may be a feature I would like to use.

So my question is, does this look like expected behaviour or is this an issue?

Thanks
Steve Hindmarch,
BT’s Global Division
This email contains information from BT, that might be privileged or 
confidential. And it's only meant for the person above. If that's not you, 
we're sorry - we must have sent it to you by mistake. Please email us to let us 
know, and don't copy or forward it to anyone else. Thanks.
We monitor our email systems and may record all our emails.


British Telecommunications plc., 81 Newgate Street London EC1A 7AJ
Registered in England no: 180

Unexpected Behaviour In LookupRecord With "Route To success" Strategy

[stephen.hindmarch]

Hi all,

I am trying to set up a simple enrichment pipeline where flow records get 
enriched from a Redis Distributed Map cache and I use a sequence of 
LookupRecord processors to gather the enrichment data. I am using the "Route to 
success" routing strategy because I would like to avoid fragmenting my flow 
files. However, the results are not what I expected and if the first record 
does not match an enrichment record then no records get enriched.

Here is a simple test case I have created.

1: Create a lookup record processor with the following parameters
Result RecordPath = /mood
Routing Strategy = Route To Success
key = concat('mood/',name)

2: Add these keys to my Redis index.
set mood/fred happy
set mood/charlie sad

3: Send in this flow file
[{"name":"fred"},{"name":"bill"},{"name":"charlie"}]

4: View the result
[{"name":"fred","mood":"happy"},{"name":"bill","mood":null},{"name":"charlie","mood":"sad"}]

That looks OK, every lookup has happened, and I can see that Bill was not 
matched as the enriched value is null.

5: Now try a different flow file, with Bill first.
[{"name":"bill"},{"name":"fred"},{"name":"charlie"}]

Result
[{"name":"bill"},{"name":"fred"},{"name":"charlie"}]

So because the first record did not match, no matches are made, and it looks as 
if the processing never happened.

6: Change the routing strategy to "Route to matched/unmatched". The result is
Matched => [{"name":"fred","mood":"happy"},{"name":"charlie","mood":"sad"}]
Unmatched => [{"name":"bill"}]

So I have achieved all of my lookups, but the cost is I have fragmented my flow 
file. After 4 lookups my original flow file (which in production will have a. 
1000 records) will have been fragmented into 16 separate files, with a 
consequent impact on performance. Also the indication that the unmatched record 
was not matched is lost, which may be a feature I would like to use.

So my question is, does this look like expected behaviour or is this an issue?

Thanks
Steve Hindmarch,
BT's Global Division
This email contains information from BT, that might be privileged or 
confidential. And it's only meant for the person above. If that's not you, 
we're sorry - we must have sent it to you by mistake. Please email us to let us 
know, and don't copy or forward it to anyone else. Thanks.
We monitor our email systems and may record all our emails.

British Telecommunications plc., 81 Newgate Street London EC1A 7AJ
Registered in England no: 180

Return JSON From Redis Lookup

[stephen.hindmarch]

Hi all,

I am trying to enrich a JSON record payload with some extra JSON stored on a 
Redis server by doing a LookupRecord, backed up with a 
DistibutedMapCacheLookupService, RedisDistributionMapCacheClientService and a 
RedisConnectionPoolService.

The JSON is stored as a string which I load into the data base with the pipe 
protocol, like this.

*3
$3
SET
$10
protocol.6
$79
{"name":"tcp","code":6,"alias":"TCP","comment":"transmission control protocol"}

And from the command line it looks like this.

get protocol.6
"{\"name\":\"tcp\",\"code\":6,\"alias\":\"TCP\",\"comment\":\"transmission 
control protocol\"}"

So already I am a bit on edge as this is getting escaped. I configure the 
lookup with the following parameters.


  *   Result RecordPath = /Enrichment/Network/Transport
  *   Routing Strategy = Route To Success
  *   Record Result Contents = Insert Entire Record
  *   Record Update Strategy = Use Property
  *   key = concat('protocol.',/network/iana_number)
Where each input record includes an integer already in the field 
"/network/iana_number".

The first thing I notice is that if the path "/Enrichment/Network/Transport" 
does not already exist in the record then the result will not create the right 
path, but instead my result will get put somewhere like 
"/Enrichment/Transport". So further upstream I have to include the creation of 
that path in a Jolt transform.

The second thing is that the return value is a String, not JSON, and looks like 
this.

"Enrichment" : {
  "Network" : {
"Transport" : 
"{\"name\":\"tcp\",\"code\":6,\"alias\":\"TCP\",\"comment\":\"transmission 
control protocol\"}"
  }
}

So I have the escaped string still, not the JSON record I was hoping for. I can 
try an UpdateRecord with this transformation:

/Enrichment/Network/transport = unescapeJson(/Enrichment/Network/Transport)

and look at the results side by side.

"Enrichment" : {
  "Network" : {
"Transport" : 
"{\"name\":\"tcp\",\"code\":6,\"alias\":\"TCP\",\"comment\":\"transmission 
control protocol\"}",
"transport" : "{name=tcp, code=6, alias=TCP, comment=transmission control 
protocol}"
  }
}

So it is still a string, and not a record. I have a feeling I lost battle at 
the Redis storage side; once the server was adamant it was a String there was 
nothing else I could do, but I was wondering what else I could try. I have 
tried various ways to remove the escaping but none resolve my issue. I was 
wondering if anybody else had seen this sort of problem and had resolved it 
successfully.

I could probably move forward with some regexes but that seems to contradict 
the point of passing JSON around in the first place.

Thanks in advance.

Steve Hindmarch

RE: QueryRecord with Union type

[stephen.hindmarch]

Mark,

I am using a Docker image of apache/nifi:latest which I pulled about 2 months 
ago. Image ID 0070c5c5d9fd. From the UI it says tag 1.15.3-RC1 and commit 
753c311. I am guessing it must be this: GitHub - apache/nifi at 
nifi-1.15.3-RC1.

Just to let you know, I did come to a satisfactory solution that does a 
convert, normalise, filter and coerce to a new schema using only two processors 
and two services (plus a schema registry). I first use a query record which if 
I work with the integer values means I have “select * from flowfile where 
flag_s = 1”. Doing the filtering early feels like a better idea, as I am not 
processing records I later discard. By using a CSV reader and a JSON writer I 
get the conversion from CSV to JSON for free. And by working with the integers 
I have no need for the union.

Then I use a JOLT transform to coerce the JSON into the form of the new schema. 
At the same time I can switch the int values to booleans by using a 
modify-overwrite. Here is the step, operating on the flag in its new position 
(“Network.flags.s”), which uses a trick of having the integer value pick the 
boolean out of a fixed array.

{
   "operation":"modify-overwrite-beta",
   "spec": {
 "*": {
   "Network": {
 "flags": {
   "*": "=elementAt(@(0),false,true)"
 }
   }
 }
   }
}

Regards,
Steve Hindmarch

From: Mark Payne 
Sent: 18 March 2022 14:44
To: users@nifi.apache.org
Subject: Re: QueryRecord with Union type

Steve,

What version of nifi are you running? I’d tried that on the latest “main” 
branch and it worked as expected.

Thanks
-Mark


On Mar 18, 2022, at 5:49 AM, 
stephen.hindma...@bt.com wrote:

Mark,

Thank you for your response. I thought that was probably the case, but I tried 
a cast and it did not work. I got this error.

Query:
select *
from flowfile
where cast(flag_s as boolean) = true

Error:
org.apache.calcite.sql.validate.SqlValidatorException: Cast function cannot 
convert value of type JavaType(class java.lang.Object) to type BOOLEAN

By taking the union out of the input schema I could get the query to work, but 
I did find myself getting tangled up in managing various schemas so I am trying 
to use infer/inherit read/write services instead. I have inherited a very 
complex flow from a team that have long departed and am looking to simplify it 
to improve performance and maintainability. I need to convert from CSV/TSV to 
JSON, normalise fields, filter unwanted records, enrich with more JSON and 
finally publish to a customer defined schema, so I do need a few steps along 
the way. I am exploring each step in order to validate my redesign so I take 
your point about minimising the number of processes and will look again at 
combining steps in the query process, although I am also a fan of the JOLT 
transform as I have used that often in previous projects.

Regards
Steve Hindmarch

From: Mark Payne mailto:marka...@hotmail.com>>
Sent: 17 March 2022 14:17
To: users mailto:users@nifi.apache.org>>
Subject: Re: QueryRecord with Union type

Steve,

Because your schema has a union, the SQL engine doesn’t really know how to 
interpret the data. So it interprets it as a “Java Object.” Essentially,
it could be anything. But you can’t compare just anything to true - you need to 
compare a boolean to true. So you need to tell the SQL engine that the
value you’re looking at is, in fact, a boolean.

You can do that with a simple CAST() function in your SQL:

SELECT *
FROM FLOWFILE
WHERE CAST(flag_s AS BOOLEAN) = true

That should give you what you’re looking for.

Also worth nothing - you mentioned that you’re using ConvertRecord and 
UpdateRecord before QueryRecord.
99% of the time, you should not be using ConvertRecord in conjunction with any 
other Record processor. Because the Record processors like UpdateRecord
allow you to use any Record Reader, it doesn’t make sense to convert the data 
first using ConvertRecord - it’s just extra overhead.
And, in fact, you may be able to eliminated the UpdateRecord, as well, as just 
use the SQL within QueryRecord to perform the transformation needed on the fly,
rather than having another step to update the data, which requires reading the 
data, parsing it, updating it, serializing the data, writing the data. This may 
not
be possible, depends on what you’re updating. But QueryRecord does support 
RecordPath expressions so it’s worth considering.

Thanks
-Mark





On Mar 15, 2022, at 8:35 AM, 
stephen.hindma...@bt.com wrote:

I am having a play with QueryRecord to do some filtering but I have run across 
this problem. I have a schema for my records which includes a union type, so 
the relevant part of the schema is

{
  "type":"record",
  "namespace":"blah",
  "name":"SimpleTraffic",
  "fields":[
{"name":"src_address","type":"string"},
{"name":"flag_s","type":["int","boolean"]}
  ]
}

RE: QueryRecord with Union type

[stephen.hindmarch]

Mark,

Thank you for your response. I thought that was probably the case, but I tried 
a cast and it did not work. I got this error.

Query:
select *
from flowfile
where cast(flag_s as boolean) = true

Error:
org.apache.calcite.sql.validate.SqlValidatorException: Cast function cannot 
convert value of type JavaType(class java.lang.Object) to type BOOLEAN

By taking the union out of the input schema I could get the query to work, but 
I did find myself getting tangled up in managing various schemas so I am trying 
to use infer/inherit read/write services instead. I have inherited a very 
complex flow from a team that have long departed and am looking to simplify it 
to improve performance and maintainability. I need to convert from CSV/TSV to 
JSON, normalise fields, filter unwanted records, enrich with more JSON and 
finally publish to a customer defined schema, so I do need a few steps along 
the way. I am exploring each step in order to validate my redesign so I take 
your point about minimising the number of processes and will look again at 
combining steps in the query process, although I am also a fan of the JOLT 
transform as I have used that often in previous projects.

Regards
Steve Hindmarch

From: Mark Payne 
Sent: 17 March 2022 14:17
To: users 
Subject: Re: QueryRecord with Union type

Steve,

Because your schema has a union, the SQL engine doesn’t really know how to 
interpret the data. So it interprets it as a “Java Object.” Essentially,
it could be anything. But you can’t compare just anything to true - you need to 
compare a boolean to true. So you need to tell the SQL engine that the
value you’re looking at is, in fact, a boolean.

You can do that with a simple CAST() function in your SQL:

SELECT *
FROM FLOWFILE
WHERE CAST(flag_s AS BOOLEAN) = true

That should give you what you’re looking for.

Also worth nothing - you mentioned that you’re using ConvertRecord and 
UpdateRecord before QueryRecord.
99% of the time, you should not be using ConvertRecord in conjunction with any 
other Record processor. Because the Record processors like UpdateRecord
allow you to use any Record Reader, it doesn’t make sense to convert the data 
first using ConvertRecord - it’s just extra overhead.
And, in fact, you may be able to eliminated the UpdateRecord, as well, as just 
use the SQL within QueryRecord to perform the transformation needed on the fly,
rather than having another step to update the data, which requires reading the 
data, parsing it, updating it, serializing the data, writing the data. This may 
not
be possible, depends on what you’re updating. But QueryRecord does support 
RecordPath expressions so it’s worth considering.

Thanks
-Mark




On Mar 15, 2022, at 8:35 AM, 
stephen.hindma...@bt.com wrote:

I am having a play with QueryRecord to do some filtering but I have run across 
this problem. I have a schema for my records which includes a union type, so 
the relevant part of the schema is

{
  "type":"record",
  "namespace":"blah",
  "name":"SimpleTraffic",
  "fields":[
{"name":"src_address","type":"string"},
{"name":"flag_s","type":["int","boolean"]}
  ]
}

This is because I am processing CSV records that look this, where 1 is true and 
0 is false.

192.168.0.1,1

Into JSON that looks like this, using a ConvertRecord and an Update Record.

{"src_address":"192.168.0.1","flag_s":true}

Then I create a QueryRecord so I can filter out the cases where the flag is 
false. So I use this query.

select * from flowfile where flag_s = true

But I get this error

org.apache.calcite.sql.validate.SqlValidatorException: Cannot apply '=' to 
arguments of type ' = '

Is this because the type is a Union type and the Calcite processor cannot work 
out which subtype it should be? Can I do anything to persuade the query to use 
an operator or a function on this field to make it usable? I have tried casting 
to Boolean or Char but no success. Or do I need to use two separate “before” 
and “after” schemas to eliminate the union?

Regards

Steve Hindmarch

QueryRecord with Union type

[stephen.hindmarch]

I am having a play with QueryRecord to do some filtering but I have run across 
this problem. I have a schema for my records which includes a union type, so 
the relevant part of the schema is

{
  "type":"record",
  "namespace":"blah",
  "name":"SimpleTraffic",
  "fields":[
{"name":"src_address","type":"string"},
{"name":"flag_s","type":["int","boolean"]}
  ]
}

This is because I am processing CSV records that look this, where 1 is true and 
0 is false.

192.168.0.1,1

Into JSON that looks like this, using a ConvertRecord and an Update Record.

{"src_address":"192.168.0.1","flag_s":true}

Then I create a QueryRecord so I can filter out the cases where the flag is 
false. So I use this query.

select * from flowfile where flag_s = true

But I get this error

org.apache.calcite.sql.validate.SqlValidatorException: Cannot apply '=' to 
arguments of type ' = '

Is this because the type is a Union type and the Calcite processor cannot work 
out which subtype it should be? Can I do anything to persuade the query to use 
an operator or a function on this field to make it usable? I have tried casting 
to Boolean or Char but no success. Or do I need to use two separate "before" 
and "after" schemas to eliminate the union?

Regards

Steve Hindmarch

RE: Records - Best Approach to Enrich Record From Cache

[stephen.hindmarch]

Redis does allow multiple gets in the one hit with MGET. If you search for all 
keys the response is an ordered list of matching values, with null in place 
where there is no match.

Steve Hindmarch

From: Nick Lange 
Sent: 07 March 2022 04:46
To: users@nifi.apache.org
Subject: Records - Best Approach to Enrich Record From Cache

HI all -
 I have a record set of objects that each need enrichment of about 10/20 fields 
of data from a Redis Cache. In a perfect world, I'd hit the cache once and 
return a json blob for further extraction  - ideally in a single hop.  I don't 
see an easy way to do this with the record language, but perhaps I've missed 
something.

Lacking any better sophistication, I'm currently doing this brute-force with 
10-20 hits to the cache for each field. I'm hoping that the mailing list has 
better suggestions.

Thank you
Nick