PutRecord sinks
Looking for feedback on if 2 new sinks would be accepted by the community. 1. InvokeHttpSink: that would make an http request per record, to insert or update a remote end point 2. DistributredMapCacheSink: to insert records into the cache map by a key given a record path
Re: [DISCUSS] Release of Apache NiFi 1.13.0
I wouldn't mind https://github.com/apache/nifi/pull/4554 making it in. Have seen a couple more instances where people have created custom processors to work around this PR. On Thu, Nov 26, 2020 at 11:10 AM Mike Thomsen wrote: > Also, for most of us (committers and community members), it's a > holiday season so might be good to delay a release to early January > anyway. Just a thought. > > On Thu, Nov 26, 2020 at 11:08 AM Mike Thomsen > wrote: > > > > https://github.com/apache/nifi/pull/4638 - This is a big increase in > > graph functionality, and I want to give him a chance to get his first > > contribution in with 1.13 (we may have some more of his work we can > > open source). > > > > Matt's also reviewing some of my Cassandra and graph tickets as well. > > I wouldn't call those **blockers**, but having the Cassandra DMC fully > > fleshed out would be nice. > > > > On Thu, Nov 26, 2020 at 9:10 AM Otto Fowler > wrote: > > > > > > NIFI-7795 https://github.com/apache/nifi/pull/4656 > > > NIFI-7761 https://github.com/apache/nifi/pull/4513 > > > NIFI-2072 https://github.com/apache/nifi/pull/4384 > > > > > > From: Pierre Villard > > > > > > Reply: dev@nifi.apache.org > > > Date: November 26, 2020 at 04:55:56 > > > To: dev@nifi.apache.org > > > Subject: [DISCUSS] Release of Apache NiFi 1.13.0 > > > > > > Hi community, > > > > > > Starting a discussion thread around the release of NiFi 1.13.0. We > added > > > quite a lot of significant improvements and features since the release > of > > > 1.12.1 - I see 143 JIRAs with 1.13.0 as the fix version. I think it > makes > > > sense to consider a new release. > > > > > > Please share here the JIRAs with opened pull requests that we would > need to > > > look at to make this release happen. > > > > > > Thanks, > > > Pierre >
Re: Parameter Contexts Expression Language scopes
I also see this as a good idea, if only for transportability. If the dev
environment is in a docker container and wants to use an environment
variable to set the context, but in production where the sensitive values
need more protection they can be set manually or in another way.
Chris Sampson's idea of notifying the users if a value changes is a good
one, and likely enough to handle the "inconsistency" factor if someone
changes the environment variable.
Phillip
On Mon, Oct 19, 2020 at 5:27 PM u...@moosheimer.com
wrote:
> If parameters make it possible to use environment variables, then the
> system can be configured in one place. You would not separate between
> expression language and parameters.
>
> It is also possible that an environment variable should be replaced by a
> parameter or vice versa.
> It makes sense to configure this in a central place.
>
> Therefore I think this is a good suggestion.
>
> Mit freundlichen Grüßen / best regards
> Kay-Uwe Moosheimer
>
> > Am 19.10.2020 um 23:18 schrieb Bryan Bende :
> >
> > I think there is some confusion based on terminology...
> >
> > A given property has an expression language scope defined, which can
> > be "flow file attributes" or "variable registry only". This was
> > created mostly for documentation purposes so that users could look at
> > a property in the docs and see "expression language supported: true"
> > and know which values they could reference. What it really comes down
> > to in the code is the difference between
> > "someProperty.evaluateAttributeExpressions()" and
> > "someProperty.evaluateAttributeExpressions(flowFile)"... meaning can I
> > reference a value from a flow file or not.
> >
> > In the case of "variable registry only", the value can come from any
> > of the following...
> > a) system properties
> > b) expression language
> > c) process-group variable registry
> > d) file-based variable registry
> >
> > When people have stated that "variables are being deprecated in favor
> > of parameters", they are referring to the last two items (c & d).
> >
> > The reason being that parameters solved several short-comings of those
> > two options...
> >
> > - the ability to store sensitive values encrypted
> > - the ability to reference them from any property using a new syntax
> > #{...}, not long dependent on component developer saying
> > "expressionLanguageSupported(true)" on the descriptor
> > - the ability to create policies for which users/groups could
> > reference a set of parameters
> > - the hierarchical ambiguity of what "${foo}" actually resolves to
> >
> > If you just want to use environment variables, why use parameter
> > contexts? Expression language has always offered access to env vars
> > and still will going forward.
> >
> > The one argument I can see is that not all properties support
> > expression language, so using parameters gives you a way around that.
> >
> >
> >
> >
> >
> >> On Mon, Oct 19, 2020 at 3:58 PM Chris Sampson
> >> wrote:
> >>
> >> Being based primarily in Docker containers and having experience with
> both
> >> Kubernetes (where secrets such as KESTORE_PASSWORD can be injected as
> >> environment variables or files) and Docker Swarm (which only handles
> >> secrets as environment variables), I'd have definitely been wanting this
> >> before moving from Variables to Parameters if I was still in Swarm (or
> >> Docker Compose/straight up Docker).
> >>
> >> It's certainly possible to script creating/updating Parameters via the
> >> Toolkit/NiPyAPI, but in Docker Swarm that's not so easy (whereas it's
> >> possible as a Job in Kubernetes, for example). So environment variables
> >> could save the day in that instance.
> >>
> >> I guess one likely problem (but no different to how I guess the Variable
> >> Registry uses env vars) would be how NiFi will handle changes to the env
> >> vars - does it:
> >>
> >> - ignore them until instance restart, which could lead to maintainer
> >> confusion (I've changed KEYSTORE_PASSWORD in the env but things are
> still
> >> failing in NiFi)
> >> - alert the maintainer to the fact that the env var has changed and a
> >> Parameter needs updating, with the new value being used after all
> >> associated processors/controllers have been restarted
> >> - automatically attempt to update the parameters by restarting all
> >> associated processors/controllers, which I'd guess would be a bit
> dangerous
> >> for interrupting in-flow data, etc.
> >>
> >>
> >> ---
> >> *Chris Sampson*
> >> IT Consultant
> >> chris.samp...@naimuri.com
> >>
Re: request to review NIFI-7738, Provenance Search Events - add reverse query capability
Pierre, I think this discussion brings up a valid conversation point. At some point a PMC member needs to approve the merge request, so from a contributors level what can we do to make that merge both easier and/or more likely to happen. That and how the community can help filter down the ever growing backlog of PRs. Thanks and look forward to helping, Phillip On Wed, Oct 7, 2020 at 2:04 PM Pierre Villard wrote: > Nissim, > > Thanks a lot for your contribution but there are 277 pull requests opened > at this time. This is a community effort, and if you expect people to > review your PRs, you'd have to also try reviewing PRs opened by others in > the community. Otherwise this will need to wait for someone with some > bandwidth to focus on this. > > Thanks, > Pierre > > Le mer. 7 oct. 2020 à 19:59, Nissim Shiman a > écrit : > > > Second attempt... > > > > > > Hello NiFi team, > > > > Could someone respond to this email. > > > > Thanks, > > Nissim Shiman > > On Monday, October 5, 2020, 11:01:57 AM EDT, Nissim Shiman > > wrote: > > > > Hello NiFi team, > > Could someone review the pull request for NIFI-7738 ( > > https://issues.apache.org/jira/browse/NIFI-7738), that was put in 5 days > > ago? > > (Provenance Search Events - add feature to allow inverse query) > > > > One case this will come in handy is if a particular flowfile has an issue > > and goes in circles through processors and ends up emitting a lot of > > provenance making it hard to follow other flowfiles. > > > > This feature will allow users to query for all provenance except for the > > uuid of the problematic flowfile. > > > > > > > > This can also be used for any indexed field or attribute as well. > > This has been compiled and tested on java 8 and java11 for all 4 > > provenance repository implementations for both indexed fields and > indexed > > attributes > > The only "real" code changes were made to LuceneUtil.java (to handle > > WriteAhead, EncryptedWriteAhead and Persistent Provenance Repostories) > and > > VolatileProvenanceRepository.java > > The rest are just involved in propagating the ability to query the "NOT" > > option for a given user entered provenance search value from the gui to > the > > backend query. > > > > Thank You, > > Nissim Shiman >
Incorrect data provenance datetime
I am running a 1.11.4 instance with adoptopenjdk 1.11. The timestamps on the data provenance events are off by ~2 months and the date on the system is correct. Thanks for any help, Phillip
Using ValidateRecord to validate schemas
Would it be reasonable to add the details of the failures to the flow file attributes? I know they exist on the provenance event, but that can not be persisted easily to file for later analysis and correction. This also seems harder to locate for non-developers and flow designers. Thanks, Phillip
Re: question about Nifi Custom processor
So a couple notes based on your debug statements.
"2.1 -> 2.2" If you do not plan to provide an input to this processor and
run it on cron or timer, you can just do the session.create at the top.
This will create a new flowfile. If you do expect an incoming flow file
just return in the null check and the next trigger will try the get() again
until it receives a flowfile.
"3 -> 3.1" You are not providing the actual string, you are passing a
static value of "JSON_PATH". I would do a "final String jsonPath =
context.getProperty(JSON_PATH);" around comment 2.3 and then use jsonPath
in your FileReader.
Object obj = new JSONParser().parse(new FileReader("JSON_PATH"));
Hope this helps,
Phillip
On Tue, Jul 23, 2019 at 10:13 PM Farooq Mustafa
wrote:
> Hello,
>
>
> I am writing my first custom processor. I followed the example here
> https://www.nifi.rocks/developing-a-custom-apache-nifi-processor-json/
>
>
> In the settings, I have /tmp/jsoniput/sample1.json file
>
>
> The source code;
>
> public void onTrigger(final ProcessContext context, final ProcessSession
> session) throws ProcessException {
>
> final AtomicReference value = new AtomicReference<>();
> getLogger().warn("sample - Farooq - 1 , value = <" + value.toString() +
> ">");
>
> FlowFile flowfile = session.get();
> getLogger().warn("sample - Farooq - 2");
>
> if ( null == flowfile) {
> getLogger().warn("sample - Farooq - 2.1 - flowfile is NULL *** ");
> flowfile = session.create();
> flowfile = session.get();
> getLogger().warn("sample - Farooq - 2.2 - session.create() /
> session.get() --- NOT SURE ");
> }
> getLogger().warn("sample - Farooq - 2.3 JSON_PATH :" +
> context.getProperty(JSON_PATH).getValue());
>
> session.read(flowfile, new InputStreamCallback() {
> @Override
> public void process(InputStream in) throws IOException {
>
> try {
> getLogger().warn("sample - Farooq - 3 JSON_PATH :" +
> context.getProperty(JSON_PATH).getValue());
> Object obj = new JSONParser().parse(new FileReader("JSON_PATH"));
> getLogger().warn("sample - Farooq - 3.1 ");
>
>
>
> The log output:
>
> 2019-07-20 13:58:42,625 WARN [Timer-Driven Process Thread-4]
> o.a.nifi.processors.sitc.JsonProcessor
> JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] sample - Farooq - 1
> , value =
>
> 2019-07-20 13:58:42,625 WARN [Timer-Driven Process Thread-4]
> o.a.nifi.processors.sitc.JsonProcessor
> JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] sample - Farooq - 2
>
> 2019-07-20 13:58:42,625 WARN [Timer-Driven Process Thread-4]
> o.a.nifi.processors.sitc.JsonProcessor
> JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] sample - Farooq -
> 2.1 - flowfile is NULL ***
>
> 2019-07-20 13:58:42,625 WARN [Timer-Driven Process Thread-4]
> o.a.nifi.processors.sitc.JsonProcessor
> JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] sample - Farooq -
> 2.2 - session.create() / session.get() --- NOT SURE
>
> 2019-07-20 13:58:42,625 WARN [Timer-Driven Process Thread-4]
> o.a.nifi.processors.sitc.JsonProcessor
> JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] sample - Farooq -
> 2.3 JSON_PATH :/tmp/jsoninput
>
> 2019-07-20 13:58:42,626 ERROR [Timer-Driven Process Thread-4]
> o.a.nifi.processors.sitc.JsonProcessor
> JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267]
> JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] failed to process
> session due to java.lang.NullPointerException; Processor Administratively
> Yielded for 1 sec: java.lang.NullPointerException
>
> java.lang.NullPointerException: null
>
> at
> org.apache.nifi.controller.repository.StandardProcessSession.getRecord(StandardProcessSession.java:574)
>
> at
> org.apache.nifi.controller.repository.StandardProcessSession.validateRecordState(StandardProcessSession.java:3132)
>
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2187)
>
>
> And the basic flow:
> [image: image.png]
>
>
>
> Any ideas or guidance will be appreciated.
>
>
> Thanks
>
> Farooq
>
