Ryan I am not entirely sure I fully understand the scenario so read the following with a bit of caution.
But the gist I think i'm reading is that system A generates data against a given schema referenceable with some guid say 'GUID1'. For every GUID1 from System A there is a similar/but possibly different schema in System B referenceable again by some guid say GUID1_B. Are those base schemas you referenced say GUID1 and GUID1_B compatible in that all fields of import in GUID1 schema will be in GUID1_B schema? As-in can you treat GUID1_B schemas as a superset of GUID1 schemas? In any event, it is very possible that you can achieve this using the Record processors and Record oriented controller services. You can 'read' data from system A in NiFi using record readers that reference schemas from systemA and 'write' data using system B schemas in NiFi. The record oriented processors combined with their pluggable record readers and writers allows this to happen. The schema for the reader and writer can be different and the processors will map things over by field names/types. That leaves the problem of 'how to access' the schema information so NiFi knows what to do during read/write phases. You can either ensure all these schemas are in the nifi schema registry and can be looked up by some well established name and naming scheme. You could also do it via a SQL lookup if you have the schemas in some database (similar to what Carlos might be suggesting but without Jolt). Or some other way.. Now, if your mapping logic from System A schema to System B schema is more complex then you'll want something like JOLT most likely to help manage those transforms. You can also do some routing on the requests from System A to specialized JOLT mappers for each case of converting to System B schemas. That will end up in a lot of config but it is likely you can paramaterize this well using versioned flows and expression langauge statements in the variable registry. Thanks Joe On Thu, Oct 4, 2018 at 1:35 PM Ryan H <ryan.howell.developm...@gmail.com> wrote: > > Hi All, > > I have been working on an integration between two systems with NiFi in the > middle as the integration point. Basically when an event happens on System-A, > such as a record update, those changes need to be propagated to System-B. For > the first use case, I have set up a data flow that listens for incoming > requests from System-A, performs a mapping, the sends the mapped data to > System-B. > > Generalized Flow for "Create_Event" (dumbed down significantly): > System-A "Create_Event" -> HandleHTTPRequest -> JoltTransformJSON -> > InvokeHTTP -> System-B "Create_Event" > > This works great for the first case with a predefined mapping in > JoltTransformJSON. Now I want to generalize it so that the same data flow can > be used for all Create_Event's on System-A. > > Here is where the issue comes in. There is a base schema for System-A that > has a base mapping to the base schema in System-B. Users of the System have > the ability to "extend" the base schema to add/remove fields and modify the > base mapping. So each time the Create_Event happens, the mapping that is used > should be the unique mapping spec associated to that user (call it a GUID > that comes along with the request). > > The data flow is the exact same for all Create_Events, except for the > mapping, which will be unique to the user. > > Does anyone know of a way to load up a different mapping to be used on a > per-request basis? I used JoltTransformJSON just as a proof of concept, so it > does not need to be a Jolt spec and can be modified to meet the needs of > whatever would work for this. > > I started to look into schema registry, but kind of got lost a bit and wasn't > sure if it could be applied to this situation. > > Any Help is, as always, greatly appreciated. > > > Cheers, > > Ryan H.
