Hi Rogier, Thanks for your message. This is an interesting use case. In a way it inverts the typical use of NiFi, which is where the flow files are the data being moved and the flow logic is in the flow definition / processor config. Instead this puts the parameters of the job/workflow into the flowfile, which presumably gets enhanced as it moves through the flow so you end up with one object/document containing your workflow parameters and data/results. Is that accurate?
I understand correctly, this sounds like a workflow orchestration problem, which is similar to but has some subtle differences from data flow management. There are tools that try to solve workflow orchestration. Two that come to mind are Conductor [1] and Cadence [2]. NiFi can do this, and I see plenty of flows that use flowfile attributes to store some control signals or values needed for flow logic. But because its not the core use case, I think NiFi developers / extension authors don't think of it when building components, which is why they don't think of enabling expression language on certain properties. This isn't really a response to whether NiFi should / should not add broader expression language support to properties nor is it an opinion on wheterh NiFi should or should not try to serve the needs of workflow orchestration / job execution. Others on this list may have opinions on that. I'm just offering my perspective on why this isn't already the case. AFAIK, in many cases adding EL support to processor properties is a fairly straightforward effort; the challenge, as you point out, is applying it broadly to all our existing processors (and new processors as they get developed) rather than just one or two. [1] https://conductor.netflix.com/ [2] https://cadenceworkflow.io/ Cheers, Kevin On Oct 18, 2022 at 06:05:14, TIMMERMANS Rogier < rogier.timmerm...@contractor.voo.be> wrote: > Hello, > > > > Apologies in advance if this is the wrong list to send this type of query > to. > > After short discussion with Chris Sampson on > https://issues.apache.org/jira/browse/NIFI-8214 he proposed to send out > email to this list; I hope it finds you well. > > > > We (several of my colleagues) sometimes use a pattern where we build a > nifi flow that gets initiated by a short json configuration file; the > initial input file (or generated flowfile) contains simple configuration > data for the rest of the flow and sets up things like Remote paths, users, > testcase IDs, endpoints to hit, etc… as a file is easier to manage, > maintain and archive than a graphical set of linked components with logic. > > This setup allows relative easy building of a generalized graphical logic > flow with (less issue below) minimal branching as ‘controlling where stuff > goes or talks to’ is effectively on the input file. > With some processing and splitting upfront each flowfile will effectively > go down it’s intended path and carry the needed values as attributes for > downstream usage where they act as dynamic configuration/parameters (or > whatever terminology you wish to apply) and we are also able to use > calculations/logic to pick output locations, etc. To make it more concrete > just imagine a use case where a simple attribute value can decide using a > production system or a test system endpoint where all logic is identical > but the only difference will be an input and/or output location. Granted > the example is simple but it does demonstrate how this can get out of hand > with multiple branching locations and duplicated components. > > We do face the issue that several NiFi components do not support > Attributes (or expression language) on several key configuration properties > (some typical example components with this limitation: FTP components, > ElasticPut, and there surely will be a few others based off the similar > base-classes which I haven’t used/found yet); this forces us into building > dedicated routes+ duplicated pattern of components because a simple > destination URI or an Remote Input/Output path cannot be dynamically > adjusted but only be set through more pre-defined constants (parameters or > through VarRegistry). > This reduces flexibility quite a bit and introduces a lot of complexity to > the graphical view as this obviously means more graphical clutter on the > worksheet, a lot more connection lines, additional branching, etc… > A minor change in logic or a property needs to be touched in several > places making this relatively error-prone to maintain as well. > > > > I’d like to propose to extend – by default – fields like: Server > endpoints, ports , Remote Paths, Input Paths (all these similar type of > fields which are limited in several components) - with a default capability > of using attributes/expression language (and by extension parameters and > the variable registry); this will increase flexibility for many components > and could de-dupe a lot of graphical flow logic and components. > > > > > > > > Best Regards, > > > > <http://www.voo.be/> > > *Rogier Timmermans* > > *Lead Engineer VOD & OTT* > > 46 Rue Jean Jaures, B-4030 Ans > > Tel: +31 (0) 6 5428 4192 > > rogier.timmerm...@contractor.voo.be > > > > > > > Ce message transmis par voie électronique ainsi que toutes ses annexes > contiennent des informations qui peuvent être confidentielles ou protégées. > Ces informations sont uniquement destinées à l’usage des personnes ou des > entités précisées dans les champs ‘A’, ‘Cc’ et ‘Cci’. Si vous n’êtes pas > l’un de ces destinataires, soyez conscient que toute forme, partielle ou > complète, de divulgation, copie, distribution ou utilisation de ces > informations est strictement interdite. Si vous avez reçu ce message par > erreur, veuillez nous en informer par téléphone ou par message électronique > et détruire les informations immédiatement. Ce message n’engage que son > signataire et aucunement son employeur. >