Russ It sounds like what you have is a case of significant reference data you need made available to various instances of this processor that knows how to use that reference state to do its function.
This is similar to cases like IP geo enrichment where the dataset on which you'd make the decision is larger and more importantly subject to change over time. In such cases the ideal state is: (A) The reference dataset(s) is hosted at a RESTful endpoint and can be periodically pulled and stored some place local/easily accessible. (B) The processor knows where to look for this reference dataset download and is able to hot reload it on the fly to include understanding that the needed datasets might not yet be made available and it should yield until it sees them and loads them. Thanks On Wed, Sep 27, 2023 at 11:51 AM Russell Bateman <r...@windofkeltia.com> wrote: > I'm posting this plea for suggestions as I'm short on imagination here. > > We have some custom processors that need extraordinary amounts of > configuration of the sort a flow writer would have to copy and paste > in--huge amounts of Yaml, regular expressions, etc. This is what our > flow writers are already doing. It would be easier to insert a filename > or -path, but... > > ...asking a custom processor to perform filesystem I/O is icky because > of unpredictable filesystem access post installation. Thinking about how > installation is beyond my control, I don't want to make installation > messy, etc. Containers, Kubernetes deployment, etc. complicate this. > > I thought of wiring /GetFile/ to a subdirectory (problematic, but less > so?) and accepting files as input to pass on to needy processors who > would recognize, adopt and incorporate configuration based on > higher-level and simpler cues posted by flow writers as property values. > > Assuming you both grok and are interested in what I'm asking, do you > have thoughts, cautionary statements or even cat-calls to offer? Maybe > there are obvious answers I'm just not thinking of. > > Profuse thanks, > > Russ
