Take a look at the mongo lookup service. I think it could serve as a good example here. On Fri, Jan 5, 2018 at 10:49 PM Brett Ryan <brett.r...@gmail.com> wrote:
> Looking at using a lookupservice, this doesn't seem to support sending > multiple keys to the LookupService at the same time. > > What I was thinking of doing was implement a LookupService that took an > attribute "sql.query" which would use this to evaluate the query but then > pass in a map of key/value pairs for attribute-name/column-name to set the > attributes. > > I could implement this as I imagined it to work, however it will evaluate > the SQL expression multiple times for the same query on the same flow. > > I am also wondering why LookupService#getRequiredKeys must return a > Set<String>, yet; this set must only contain one value. > > > On 5 Jan 2018, at 08:32, Andy LoPresto <alopre...@apache.org> wrote: > > UpdateAttribute doesn’t pull from a database, it uses static or dynamic > attribute values and supports NiFi Expression Language. In your original > message, you didn’t mention any database interaction, so I thought you were > just trying to accomplish "I wanted to add some attributes to a FlowFile > while not altering the contents of that FlowFile”, which is indeed what > UpdateAttribute does. > > If you need to retrieve those values from a database, as Joey mentions, > the LookupService is the right tool. > > With your prior setup, the distributed map cache is as secure as the NiFi > configuration — if using secured NiFi, the communication between that node > and any other is over TLS, and within the node it’s a memory access. > > A big part of the NiFi philosophy is the same as the Unix philosophy — > each tool should do one job very well, and to perform complicated tasks, > chain those tools together. This helps with provenance reporting, usage > reporting, debugging, flow development lifecycle, maintenance, etc. A > processor which retrieves attributes from a database and updates the > incoming flowfile with them is certainly useful in the use case you > describe, but is not a generic pattern. There’s no intent to discourage > custom development, and whatever makes your flow work is fine. Just > explaining why you likely won’t see a solution like that in the NiFi > bundled components. > > > Andy LoPresto > alopre...@apache.org > *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>* > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Jan 4, 2018, at 4:07 PM, Brett Ryan <brett.r...@gmail.com> wrote: > > Thanks Andy, how would update attribute be able to get the value from sql? > > Consider a flow where a piece of information needs to be obtained from a > DB but i do not want the contents of the current FF to be altered, using > ExecuteSQL anywhere prior would not be possible due to replacing the FF > contents. > > What I had was two seperate flows, one that updates an oauth key in a db > keeping it fresh, the main flow would then read the db just before an > invokehttp. > > I was originally using a distributed map cache but had concerns that it > might not be secure and was also advised the cache server has been known to > go down. > > On 5 Jan 2018, at 04:01, Joey Frazee <joey.fra...@icloud.com> wrote: > > Andy, Brett, > > Taking a quick glance at the code it looks like it's enriching attributes > from a database according to a query. > > If that's correct, there's a LookupAttribute processor that delegates > lookups to a "LookupService" and adds attributes without altering content. > There are a variety of these LookupServices included. I think what you > implemented would make sense as a LookupService and then you could just > configure the processor to use that. It could also be used with > LookupRecord then too so there'd be a double payoff. > > -joey > > On Jan 4, 2018, 8:44 AM -0800, Andy LoPresto <alopre...@apache.org>, > wrote: > Hi Brett, > > It’s great that you found it easy to write a new processor for Apache > NiFi. It is probably an indicator that we need to improve > education/evangelism/documentation, however, that you did not find > UpdateAttribute [1], which should do exactly what you were looking for. > > [1] > https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-update-attribute-nar/1.4.0/org.apache.nifi.processors.attributes.UpdateAttribute/index.html > > Andy LoPresto > alopre...@apache.org > alopresto.apa...@gmail.com > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Jan 4, 2018, at 7:03 AM, Brett Ryan <brett.r...@gmail.com> wrote: > > Hi all, having used NiFi for a couple days I wanted to add some attributes > to a FlowFile while not altering the contents of that FlowFile. > > I had suggestions to use a script processor but that just sounded like a > hack which could become a nuisance to replicate. > > Anyway, I figured I'd write a processor to do this, anyone interested you > can find it here. > > https://github.com/brettryan/nifi-drunken-bundle > > Feel free to do with it what you please. > > I've published to maven central but it will take a day to appear in the > search. > > <dependency> > <groupId>com.drunkendev</groupId> > <artifactId>nifi-drunken-nar</artifactId> > <version>1.0.0</version> > <type>nar</type> > </dependency> > <dependency> > <groupId>com.drunkendev</groupId> > <artifactId>nifi-drunken-processors</artifactId> > <version>1.0.0</version> > </dependency> > <dependency> > <groupId>com.drunkendev</groupId> > <artifactId>nifi-drunken-bundle</artifactId> > <version>1.0.0</version> > <type>pom</type> > </dependency> > > > >
