Hi Austin,
We did consider enriching records in the GeoMesa converter, but we
also need the enriched records for other destinations e.g.
ElasticSearch, hence we were keen to keep it in NiFi. As suggested, I
think I'll write a little Groovy script to merge JSON (looked up from
HBase) from NiFi
James,
A little late to the show but hopefully this is useful.
What we typically do for data enrichment is we'll use an
EvaluateJsonPath processor to pull JSON fields out into attributes under
a common key, e.g. foo.model. We then have a PutRedis processor that
grabs everything under foo and
James,
Only skimmed this, but this looks like it might provide some interesting
ideas on how to transition from Protobuf to Avro:
https://gist.github.com/alexvictoor/1d3937f502c60318071f
Mike
On Tue, Dec 18, 2018 at 3:07 PM Otto Fowler wrote:
> What would be really cool would be if you could
What would be really cool would be if you could also load the registry with
your .protos somehow, and configure using the proto names, and then just
have your registry convert them on demand
On December 18, 2018 at 15:04:30, Otto Fowler (ottobackwa...@gmail.com)
wrote:
You could implement a
You could implement a custom schema registry that converts the protos to
schema on the fly and caches.
On December 18, 2018 at 13:55:47, James Srinivasan (
james.sriniva...@gmail.com) wrote:
Yup, my example used made-up fields to keep it simple. In reality I
have between 20 and 80 fields per
Good idea - this script is pretty close:
https://community.hortonworks.com/questions/75523/processor-for-replacing-json-values-dynamically-an.html
Thanks
On Mon, 17 Dec 2018 at 18:01, Andrew Grande wrote:
>
> James,
>
> The easiest would be to merge json in a custom processor. Not easy as in no
I know you mentioned staying schema agnostic, but if you went with the
record approach then this sounds like a good fit for the HBase lookup
service.
Steps 3-5 would be using LookupRecord with an HBaseLookupService where
you lookup by row id, and put the results into the current record.
I'm not
James,
The easiest would be to merge json in a custom processor. Not easy as in no
work at all, but given your limitations with the NiFi version could be done
sooner maybe.
Andrew
On Mon, Dec 17, 2018, 9:53 AM James Srinivasan
wrote:
> Hi all,
>
> I'm trying to enrich a data stream using
Hi all,
I'm trying to enrich a data stream using NiFi. So far I have the following:
1) Stream of vehicle data in JSON format containing (id, make, model)
2) This vehicle data goes into HBase, using id as the row key and the
json data as the cell value (cf:json)
3) Stream of position data in JSON
