I have a flow performing ETL of HL7v4 (FHIR) document on their way to
indexing and storage. Custom processors perform the important
transformations. Performance of this flow is at a premium for us. At
some point along the way I want to gate off copies of raw or of
transformed FHIR records (the flow writer's choice) to a new flow (a
"subflow" of the total flow) for the purpose of validating those FHIR
records as an option.
The main ETL flow will thus not be interrupted. Also, its performance
should not be too hugely impacted by this new subflow. I have looked at
priority techniques discussed, but usually the discussion is geared more
toward a resulting order. I want to deprecate the performance of this
new subflow to avoid handicapping the main flow, ideally from almost
shutting down the subflow to allowing it equal performance with the main
ETL flow.
Are there recommendations for such a thing? As I author many custom
processors, is there something I could be doing in my code to aid this?
I want rather to put the amount of crippling into the hands of my flow
writers a) by natural, existing configuration that's a feature of most
NiFi processors and/or b) surfacing programming choices as configuration
in my custom processor's configuration. Etc.
Any comments on this are hoped for and very welcome.
(Because I wrote so many custom processors that are crucial to my flows,
I chose the NiFi developer- instead of the users list.)
