Re: Issues while configuring nifi cluster
Hello Andy, Thanks for your support, I have deleted the flow.xmlgz file and restarted both the nodes. Now the Cluster has been connected after restarting the nodes. Thanks, *Raghava Raju* Softility, Inc. | E: vbhupathir...@softilityinc.com On Wed, Nov 6, 2019 at 5:06 AM Andy LoPresto wrote: > The message about the blank sensitive properties key is important and is > informing you that the sensitive configuration values are not being > protected because that key has not been set. However, it is not blocking > any operations from occurring. > > The subsequent messages are telling you that the Flow Registry does not > contain the specific version (version 2) of one of the flows being used, so > the node cannot sync with it. Then it reports that the node cannot join the > cluster because the local flow is not the same as the cluster flow. If you > have made local changes and want to keep them, you’ll have to perform > manual synchronization on the cluster nodes to duplicate those changes. If > not, you can delete the flow.xml.gz file from the local node’s conf/ > directory and after restarting, the node will inherit the cluster flow. > > For node 2, the same message as the beginning of node 1 is occurring. You > should check that your Flow Registry is configured correctly and has the > most recent version of the flow under version control. > > > Andy LoPresto > alopre...@apache.org > *alopresto.apa...@gmail.com * > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Nov 4, 2019, at 7:27 AM, Venkata Raghava Raju Bhupathiraju < > vbhupathir...@softility.com> wrote: > > Hello Andy, > > Thanks for you response, Please find the below errors in the log file from > both the Nifi Cluster servers, > > *Nifi - 1* > > 2019-11-04 14:04:51,070 ERROR [main] > org.apache.nifi.encrypt.StringEncryptor *A blank sensitive > properties key was provided * > 2019-11-04 14:04:51,070 ERROR [main] > org.apache.nifi.encrypt.StringEncryptor * Specify a > unique key in nifi.properties* > 2019-11-04 14:04:51,070 ERROR [main] > org.apache.nifi.encrypt.StringEncryptor * for > nifi.sensitive.props.key * > 2019-11-04 14:04:51,071 ERROR [main] > org.apache.nifi.encrypt.StringEncryptor * >* > 2019-11-04 14:04:51,071 ERROR [main] > org.apache.nifi.encrypt.StringEncryptor *The Encrypt Config > Tool in NiFi Toolkit can be used to* > 2019-11-04 14:04:51,071 ERROR [main] > org.apache.nifi.encrypt.StringEncryptor * migrate the > flow to the new key* > 2019-11-04 14:04:51,071 ERROR [main] > org.apache.nifi.encrypt.StringEncryptor > > 2019-11-04 14:04:56,455 ERROR [Framework Task Thread Thread-1] > o.a.nifi.groups.StandardProcessGroup Failed to synchronize > StandardProcessGroup[identifier=b066b84a-016d-1000--af957491] with > Flow Registry because could not retrieve version 2 of flow with identifier > 091ba32b-270a-43e5-9459-2e56e2650950 in bucket > fdaf7347-5f0a-4a8b-90f6-a2ffbd1e4636 > 2019-11-04 14:09:58,043 ERROR [Reconnect to Cluster] > o.a.nifi.controller.StandardFlowService Handling reconnection request > failed due to: org.apache.nifi.controller.UninheritableFlowException: > Failed to connect node to cluster because local flow is different than > cluster flow. > 2019-11-04 14:09:58,072 ERROR [Reconnect to Cluster] > o.a.n.c.c.node.NodeClusterCoordinator Event Reported for 10.1.1.14:8081 > -- Node disconnected from cluster due to > org.apache.nifi.controller.UninheritableFlowException: Failed to connect > node to cluster because local flow is different than cluster flow. > > *Nifi - 2* > > 2019-11-04 14:04:50,672 ERROR [main] > org.apache.nifi.encrypt.StringEncryptor > > 2019-11-04 14:04:50,672 ERROR [main] > org.apache.nifi.encrypt.StringEncryptor *A blank sensitive > properties key was provided * > 2019-11-04 14:04:50,672 ERROR [main] > org.apache.nifi.encrypt.StringEncryptor * Specify a > unique key in nifi.properties* > 2019-11-04 14:04:50,672 ERROR [main] > org.apache.nifi.encrypt.StringEncryptor * for > nifi.sensitive.props.key * > 2019-11-04 14:04:50,672 ERROR [main] > org.apache.nifi.encrypt.StringEncryptor * >* > 2019-11-04 14:04:50,672 ERROR [main] > org.apache.nifi.encrypt.StringEncryptor *The Encrypt Config > Tool in NiFi Toolkit can be used to* > 2019-11-04 14:04:50,672 ERROR [main] > org.apache.nifi.encrypt.StringEncryptor * migrate the > flow to the new key* > 2019-11-04 14:04:50,
NiFi Registry - Rolling back a flow to an older version
Hi, In the Change Version section of NiFi User Guide it states that "a versioned flow can also be rollbacked to an older version" when using the NiFi Registry. In the NiFi UI I do not see that option. How do a rollback a flow to an older version and then make the older version the current version? Thanks -David
Re: NiFi Registry - Rolling back a flow to an older version
Hello, A upgrade and rollback are both done through "Change Version", going forward or backward. In order to make the current version the latest, this feature is only available in the newest releases (NiFi 1.10.0 and registry 0.5.0) where you can "Force Commit". Thanks, Bryan On Wed, Nov 6, 2019 at 10:53 AM David Natyshak wrote: > > Hi, > > In the Change Version section of NiFi User Guide it states that "a > versioned flow can also be rollbacked to an older version" when using the > NiFi Registry. In the NiFi UI I do not see that option. How do a rollback a > flow to an older version and then make the older version the current > version? > > Thanks > > -David
NiFi 1.10.0 JRE clarification...
Looking at the release notes, I see: * Apache NiFi can now be built on either Java 8 or Java 11! When built on Java 8 it can run on Java 8 or Java 11. The implications of this statement make me ask: * At https://nifi.apache.org/download.html, how is it built? JDK 11? * Can I run it using JRE 8? Or, must I download source and build it using JDK 8 myself? I apologize; I could try it myself, but I don't have a platform running JRE 8 easy to hand, so I thought I'd ask if anyone knows. Likely, anything I do will ultimately need to run on (CentOS) JRE 8, so I'm not asking idly. Thanks, Russ
Re: NiFi 1.10.0 JRE clarification...
Releases are built with Java 8 until sometime in the future when we move to Java 11 as the minimum (likely NiFi 2.0.0). On Wed, Nov 6, 2019 at 10:58 AM Russell Bateman wrote: > > Looking at the release notes, I see: > > * Apache NiFi can now be built on either Java 8 or Java 11! When built > on Java 8 it can run on Java 8 or Java 11. > > The implications of this statement make me ask: > > * At https://nifi.apache.org/download.html, how is it built? JDK 11? > * Can I run it using JRE 8? Or, must I download source and build it > using JDK 8 myself? > > I apologize; I could try it myself, but I don't have a platform running > JRE 8 easy to hand, so I thought I'd ask if anyone knows. Likely, > anything I do will ultimately need to run on (CentOS) JRE 8, so I'm not > asking idly. > > Thanks, > > Russ > >
Re: PULL ProvenanceEvent
Joe, Just to verify what you mean, You are saying that the line: flowfile = session.putAttribute(flowfile, "receiveType", "active") could be added before session.getProvenanceReporter().receive(...) to indicate a PULL. Is this correct? Thanks, Nissim On Monday, November 4, 2019, 12:50:11 PM EST, Nissim Shiman wrote: Having an attribute added indicating passive/active/query for RECEIVE and FETCH will work, but nifi attributes are stateful (i.e. they will still be on the flowfile as metadata a couple of processor steps down the flow) Maybe an option is to expand the the api for RECEIVE and FETCH for with a new parameter for passive/active/query ? (i.e. the existing message signatures, such as [1] will remain the same, but new ones will be added to handle this new parameter? [1] https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceReporter.java#L46 On Thursday, October 31, 2019, 10:10:40 PM EDT, Joe Witt wrote: These distinctions may be meaningful. Adding them as an attribute lets the meaning convey but not introduce complexity for the majority case which is the distinction isnt key. thanks On Thu, Oct 31, 2019 at 4:05 PM Nissim Shiman wrote: > Mike, > I like the QUERY type as well. Basically a more refined PULL. Very nice. > > > Part of the challenge of adding PULL as a type is that there are currently > two flavors of RECEIVEs. > RECEIVE and FETCH [1] > > So any addition of a PULL would need a second flavor of PULL to match the > case where a flowfile's contents are being overwritten as well (i.e. as > FETCH is currently doing) > > > [1] > https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceEventType.java#L42 > > > Thanks, > Nissim > > > On Wednesday, October 30, 2019, 6:41:04 PM EDT, Mike Thomsen < > mikerthom...@gmail.com> wrote: > > I like the idea of creating PULL as a type. In fact, I'd propose that > there > are three scenarios here: > > RECEIVE - Passively acquire in a sort of hand-off situation. Ex: Kafka > subscription > PULL - Direct operations to seek out and fetch something in a targeted > fashion. Ex. GetHttp > QUERY - Go looking for the data and take what matches your search. Ex. > JsonQueryElasticsearch, GetMongo, any SQL query processor, etc. > > > > On Wed, Oct 30, 2019 at 1:31 PM Nissim Shiman > wrote: > > > Joe, > > > > > > It is hard to say how much value transit URI would bring to clarify a > > RECEIVE. > > For example a RECEIVE with transit URI of https: could be either a > > GetHTTP (i.e. active) or ListenHTTP (i.e. passive) > > > > but your idea of "a metadata item specifying active vs passive" is a very > > clever way to make this work with mimimal disruptions. > > > > My understanding of this is that the current receive() calls in > > ProvenanceReporter [1] will remain the same, but news ones will be added > > with a boolean parameter reflecting if the receive is active or passive. > > This will allow the current list of Provenance Events [2] to remain the > > same. So third party/custom processors can continue working as is > > > > Does this sound like what you are thinking? > > > > > > [1] > > > https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceReporter.java#L46 > > > > [2] > > apache/nifi > > > > > > Thanks, > > > > Nissim > > On Tuesday, October 29, 2019, 12:47:40 PM EDT, Joe Witt < > > joe.w...@gmail.com> wrote: > > > > Nissim > > > > I like the idea to introduce a more refined type of event for how data is > > brought into nifi (active - PULL, passive - RECEIVE). > > > > That said it might be sufficient to simply have this distinction be on > the > > "RECEIVE" event as a metadata item specifying active vs passive. The > > protocol utilized as mentioned in the transport URI should clarify this > > though. > > > > In short - i think there is a way here that is all opt-in for existing > > users and components. > > > > Thanks > > > > On Tue, Oct 29, 2019 at 9:41 AM Nissim Shiman > > > wrote: > > > > > Adam, > > > good points... > > > I missed a step in explaining the use case where Provenance Events is > > > incomplete... > > > Where the second nifi does a GetSFTP from the *filesytem* that the > first > > > nifi is located on > > > So the second nifi currently sends a RECEIVE event, but there is no > > > corresponding SEND event from the first nifi (nor should there be) > > > If the second nifi sent a PULL event, it would be easier for a system > > > overseer to know that there should be no corresponding SEND event > > > > > > Currently send/receive works well when nifi 1 does a PostHTTP and nifi > 2 > > > does a ListenHTTP, but not in the case above. > > > > > > The ERROR case you mention is a nice point as well, although not my > > > specific issue at the moment. > > > Thanks, > > > Nissim > > > > > > > > > > > > > > > > > > On Monday, October 28, 2019,
Re: PULL ProvenanceEvent
Nissim Notionally I am saying that session.getProvenanceReporter().receive(...) should have an option to call session.getProvenanceReporter().receive(...,ACTIVE|PASSIVE) and if not specified it would be UNSPECIFIED. I dont think this needs to be on the flowfile attribute - it would go straight to the provenance event itself which is generated by the session. Thanks Joe On Wed, Nov 6, 2019 at 11:32 AM Nissim Shiman wrote: > Joe, > > Just to verify what you mean, > > You are saying that the line: > flowfile = session.putAttribute(flowfile, "receiveType", "active") > > could be added before > session.getProvenanceReporter().receive(...) > > > to indicate a PULL. Is this correct? > > Thanks, > > Nissim > > > > > > > On Monday, November 4, 2019, 12:50:11 PM EST, Nissim Shiman > wrote: > > Having an attribute added indicating passive/active/query for RECEIVE > and FETCH will work, > > but nifi attributes are stateful (i.e. they will still be on the flowfile > as metadata a couple of processor steps down the flow) > > Maybe an option is to expand the the api for RECEIVE and FETCH for with a > new parameter for passive/active/query ? > (i.e. the existing message signatures, such as [1] will remain the same, > but new ones will be added to handle this new parameter? > > [1] > https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceReporter.java#L46 > > > On Thursday, October 31, 2019, 10:10:40 PM EDT, Joe Witt < > joe.w...@gmail.com> wrote: > > These distinctions may be meaningful. Adding them as an attribute lets > the > meaning convey but not introduce complexity for the majority case which is > the distinction isnt key. > > thanks > > On Thu, Oct 31, 2019 at 4:05 PM Nissim Shiman > wrote: > > > Mike, > > I like the QUERY type as well. Basically a more refined PULL. Very > nice. > > > > > > Part of the challenge of adding PULL as a type is that there are > currently > > two flavors of RECEIVEs. > > RECEIVE and FETCH [1] > > > > So any addition of a PULL would need a second flavor of PULL to match the > > case where a flowfile's contents are being overwritten as well (i.e. as > > FETCH is currently doing) > > > > > > [1] > > > https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceEventType.java#L42 > > > > > > Thanks, > > Nissim > > > > > >On Wednesday, October 30, 2019, 6:41:04 PM EDT, Mike Thomsen < > > mikerthom...@gmail.com> wrote: > > > > I like the idea of creating PULL as a type. In fact, I'd propose that > > there > > are three scenarios here: > > > > RECEIVE - Passively acquire in a sort of hand-off situation. Ex: Kafka > > subscription > > PULL - Direct operations to seek out and fetch something in a targeted > > fashion. Ex. GetHttp > > QUERY - Go looking for the data and take what matches your search. Ex. > > JsonQueryElasticsearch, GetMongo, any SQL query processor, etc. > > > > > > > > On Wed, Oct 30, 2019 at 1:31 PM Nissim Shiman > > > wrote: > > > > > Joe, > > > > > > > > > It is hard to say how much value transit URI would bring to clarify a > > > RECEIVE. > > > For example a RECEIVE with transit URI of https: could be either > a > > > GetHTTP (i.e. active) or ListenHTTP (i.e. passive) > > > > > > but your idea of "a metadata item specifying active vs passive" is a > very > > > clever way to make this work with mimimal disruptions. > > > > > > My understanding of this is that the current receive() calls in > > > ProvenanceReporter [1] will remain the same, but news ones will be > added > > > with a boolean parameter reflecting if the receive is active or > passive. > > > This will allow the current list of Provenance Events [2] to remain the > > > same. So third party/custom processors can continue working as is > > > > > > Does this sound like what you are thinking? > > > > > > > > > [1] > > > > > > https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceReporter.java#L46 > > > > > > [2] > > > apache/nifi > > > > > > > > > Thanks, > > > > > > Nissim > > >On Tuesday, October 29, 2019, 12:47:40 PM EDT, Joe Witt < > > > joe.w...@gmail.com> wrote: > > > > > > Nissim > > > > > > I like the idea to introduce a more refined type of event for how data > is > > > brought into nifi (active - PULL, passive - RECEIVE). > > > > > > That said it might be sufficient to simply have this distinction be on > > the > > > "RECEIVE" event as a metadata item specifying active vs passive. The > > > protocol utilized as mentioned in the transport URI should clarify this > > > though. > > > > > > In short - i think there is a way here that is all opt-in for existing > > > users and components. > > > > > > Thanks > > > > > > On Tue, Oct 29, 2019 at 9:41 AM Nissim Shiman > > > > > > wrote: > > > > > > > Adam, > > > > good points... > > > > I missed a step in explaining the use case where Provenance Events is > > > > incomplete... > > > > Whe
Re: PULL ProvenanceEvent
Joe, Very nice... Thanks! Nissim On Wednesday, November 6, 2019, 1:05:09 PM EST, Joe Witt wrote: Nissim Notionally I am saying that session.getProvenanceReporter().receive(...) should have an option to call session.getProvenanceReporter().receive(...,ACTIVE|PASSIVE) and if not specified it would be UNSPECIFIED. I dont think this needs to be on the flowfile attribute - it would go straight to the provenance event itself which is generated by the session. Thanks Joe On Wed, Nov 6, 2019 at 11:32 AM Nissim Shiman wrote: > Joe, > > Just to verify what you mean, > > You are saying that the line: > flowfile = session.putAttribute(flowfile, "receiveType", "active") > > could be added before > session.getProvenanceReporter().receive(...) > > > to indicate a PULL. Is this correct? > > Thanks, > > Nissim > > > > > > > On Monday, November 4, 2019, 12:50:11 PM EST, Nissim Shiman > wrote: > > Having an attribute added indicating passive/active/query for RECEIVE > and FETCH will work, > > but nifi attributes are stateful (i.e. they will still be on the flowfile > as metadata a couple of processor steps down the flow) > > Maybe an option is to expand the the api for RECEIVE and FETCH for with a > new parameter for passive/active/query ? > (i.e. the existing message signatures, such as [1] will remain the same, > but new ones will be added to handle this new parameter? > > [1] > https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceReporter.java#L46 > > > On Thursday, October 31, 2019, 10:10:40 PM EDT, Joe Witt < > joe.w...@gmail.com> wrote: > > These distinctions may be meaningful. Adding them as an attribute lets > the > meaning convey but not introduce complexity for the majority case which is > the distinction isnt key. > > thanks > > On Thu, Oct 31, 2019 at 4:05 PM Nissim Shiman > wrote: > > > Mike, > > I like the QUERY type as well. Basically a more refined PULL. Very > nice. > > > > > > Part of the challenge of adding PULL as a type is that there are > currently > > two flavors of RECEIVEs. > > RECEIVE and FETCH [1] > > > > So any addition of a PULL would need a second flavor of PULL to match the > > case where a flowfile's contents are being overwritten as well (i.e. as > > FETCH is currently doing) > > > > > > [1] > > > https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceEventType.java#L42 > > > > > > Thanks, > > Nissim > > > > > > On Wednesday, October 30, 2019, 6:41:04 PM EDT, Mike Thomsen < > > mikerthom...@gmail.com> wrote: > > > > I like the idea of creating PULL as a type. In fact, I'd propose that > > there > > are three scenarios here: > > > > RECEIVE - Passively acquire in a sort of hand-off situation. Ex: Kafka > > subscription > > PULL - Direct operations to seek out and fetch something in a targeted > > fashion. Ex. GetHttp > > QUERY - Go looking for the data and take what matches your search. Ex. > > JsonQueryElasticsearch, GetMongo, any SQL query processor, etc. > > > > > > > > On Wed, Oct 30, 2019 at 1:31 PM Nissim Shiman > > > wrote: > > > > > Joe, > > > > > > > > > It is hard to say how much value transit URI would bring to clarify a > > > RECEIVE. > > > For example a RECEIVE with transit URI of https: could be either > a > > > GetHTTP (i.e. active) or ListenHTTP (i.e. passive) > > > > > > but your idea of "a metadata item specifying active vs passive" is a > very > > > clever way to make this work with mimimal disruptions. > > > > > > My understanding of this is that the current receive() calls in > > > ProvenanceReporter [1] will remain the same, but news ones will be > added > > > with a boolean parameter reflecting if the receive is active or > passive. > > > This will allow the current list of Provenance Events [2] to remain the > > > same. So third party/custom processors can continue working as is > > > > > > Does this sound like what you are thinking? > > > > > > > > > [1] > > > > > > https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceReporter.java#L46 > > > > > > [2] > > > apache/nifi > > > > > > > > > Thanks, > > > > > > Nissim > > > On Tuesday, October 29, 2019, 12:47:40 PM EDT, Joe Witt < > > > joe.w...@gmail.com> wrote: > > > > > > Nissim > > > > > > I like the idea to introduce a more refined type of event for how data > is > > > brought into nifi (active - PULL, passive - RECEIVE). > > > > > > That said it might be sufficient to simply have this distinction be on > > the > > > "RECEIVE" event as a metadata item specifying active vs passive. The > > > protocol utilized as mentioned in the transport URI should clarify this > > > though. > > > > > > In short - i think there is a way here that is all opt-in for existing > > > users and components. > > > > > > Thanks > > > > > > On Tue, Oct 29, 2019 at 9:41 AM Nissim Shiman > > > > > > wrote: > > > > > > > Adam, > > > > good points... >
Re: PULL ProvenanceEvent
+1 Joe - this is a good compromise to keep the original API undisturbed. On Wed, Nov 6, 2019 at 11:05 AM Joe Witt wrote: > Nissim > > Notionally I am saying that session.getProvenanceReporter().receive(...) > should have an option to call > session.getProvenanceReporter().receive(...,ACTIVE|PASSIVE) and if not > specified it would be UNSPECIFIED. > > I dont think this needs to be on the flowfile attribute - it would go > straight to the provenance event itself which is generated by the session. > > Thanks > Joe > > On Wed, Nov 6, 2019 at 11:32 AM Nissim Shiman > wrote: > > > Joe, > > > > Just to verify what you mean, > > > > You are saying that the line: > > flowfile = session.putAttribute(flowfile, "receiveType", "active") > > > > could be added before > > session.getProvenanceReporter().receive(...) > > > > > > to indicate a PULL. Is this correct? > > > > Thanks, > > > > Nissim > > > > > > > > > > > > > > On Monday, November 4, 2019, 12:50:11 PM EST, Nissim Shiman > > wrote: > > > > Having an attribute added indicating passive/active/query for RECEIVE > > and FETCH will work, > > > > but nifi attributes are stateful (i.e. they will still be on the flowfile > > as metadata a couple of processor steps down the flow) > > > > Maybe an option is to expand the the api for RECEIVE and FETCH for with a > > new parameter for passive/active/query ? > > (i.e. the existing message signatures, such as [1] will remain the same, > > but new ones will be added to handle this new parameter? > > > > [1] > > > https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceReporter.java#L46 > > > > > > On Thursday, October 31, 2019, 10:10:40 PM EDT, Joe Witt < > > joe.w...@gmail.com> wrote: > > > > These distinctions may be meaningful. Adding them as an attribute lets > > the > > meaning convey but not introduce complexity for the majority case which > is > > the distinction isnt key. > > > > thanks > > > > On Thu, Oct 31, 2019 at 4:05 PM Nissim Shiman > > > wrote: > > > > > Mike, > > > I like the QUERY type as well. Basically a more refined PULL. Very > > nice. > > > > > > > > > Part of the challenge of adding PULL as a type is that there are > > currently > > > two flavors of RECEIVEs. > > > RECEIVE and FETCH [1] > > > > > > So any addition of a PULL would need a second flavor of PULL to match > the > > > case where a flowfile's contents are being overwritten as well (i.e. as > > > FETCH is currently doing) > > > > > > > > > [1] > > > > > > https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceEventType.java#L42 > > > > > > > > > Thanks, > > > Nissim > > > > > > > > >On Wednesday, October 30, 2019, 6:41:04 PM EDT, Mike Thomsen < > > > mikerthom...@gmail.com> wrote: > > > > > > I like the idea of creating PULL as a type. In fact, I'd propose that > > > there > > > are three scenarios here: > > > > > > RECEIVE - Passively acquire in a sort of hand-off situation. Ex: Kafka > > > subscription > > > PULL - Direct operations to seek out and fetch something in a targeted > > > fashion. Ex. GetHttp > > > QUERY - Go looking for the data and take what matches your search. Ex. > > > JsonQueryElasticsearch, GetMongo, any SQL query processor, etc. > > > > > > > > > > > > On Wed, Oct 30, 2019 at 1:31 PM Nissim Shiman > > > > > > wrote: > > > > > > > Joe, > > > > > > > > > > > > It is hard to say how much value transit URI would bring to clarify a > > > > RECEIVE. > > > > For example a RECEIVE with transit URI of https: could be > either > > a > > > > GetHTTP (i.e. active) or ListenHTTP (i.e. passive) > > > > > > > > but your idea of "a metadata item specifying active vs passive" is a > > very > > > > clever way to make this work with mimimal disruptions. > > > > > > > > My understanding of this is that the current receive() calls in > > > > ProvenanceReporter [1] will remain the same, but news ones will be > > added > > > > with a boolean parameter reflecting if the receive is active or > > passive. > > > > This will allow the current list of Provenance Events [2] to remain > the > > > > same. So third party/custom processors can continue working as is > > > > > > > > Does this sound like what you are thinking? > > > > > > > > > > > > [1] > > > > > > > > > > https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceReporter.java#L46 > > > > > > > > [2] > > > > apache/nifi > > > > > > > > > > > > Thanks, > > > > > > > > Nissim > > > >On Tuesday, October 29, 2019, 12:47:40 PM EDT, Joe Witt < > > > > joe.w...@gmail.com> wrote: > > > > > > > > Nissim > > > > > > > > I like the idea to introduce a more refined type of event for how > data > > is > > > > brought into nifi (active - PULL, passive - RECEIVE). > > > > > > > > That said it might be sufficient to simply have this distinction be > on > > > the > > > > "RECEIVE" event as a metadata item specifying active vs passi
Want to be minifi committer
Hi NiFi developer team, I am a big data admin / developer at medical domain at MD Anderson Cancer Center. We have been working at data streaming for quite a while. When I do the use case in medical iot, I can see there are a lot of gaps between current minifi-nifi interaction, so I would like to join the minifi developer group and contribute my idea and efforts as a developer. I had years of development in Java before becoming a hadoop admin. I had quite some experiences coding in big data stack, including NiFi processors, Kafka and Spark. The primary language I am using is Java, but I also use Python and Scala. Anyway, can you please what is the best way to join the developer family? Looking forward to make a more robust minifi product that is trimmed toward medical IoT. Best, Daniel Wang Hadoop Admin and Big data developer MD Anderson Cancer Center
BUG - flowFileEvent combineCounters hashmap overwritten
Hello, some developer can verify this: https://issues.apache.org/jira/browse/NIFI-6853 It is related to the bug I reported a few days ago NIFI-6800 This is another problem in the StandardProcessSession.java promptly in the combination of the counters for the event repository (flowfileEvent object).
Cluster question - interval vs external routing
Hi there, We have two addresses on each of our NiFi cluster hosts. One is externally facing, which we use to access the UI, and connect to receiving processors from the wider network. We are trying to configure NiFi to use the internal network for its cluster traffic. However if I use the internal address IPs in the nifi.cluster.node.address and nifi.remote.input.host properties, I end up getting two entries for each host in the cluster (both showing the external address) and only one of them will be online and connected. The other shows as disconnected. What am I missing here? Regards, Phil
Re: Want to be minifi committer
Hi Daniel, Glad to hear about your interest in MiNiFi! MiNiFi's processes and procedures are not significantly different than those of NiFi, so the Contributor Guide [1] is a great reference point for how to engage. As you are likely aware, we have both Java and C++ implementations. Based on your listed experience, it seems the Java variant would be the best for core development, but we also have some extensions and testing code in place that is written in Python. You can find issues for our Java [2] and C++ [3] to see if there is anything you would like to work on. You mentioned some gaps that you would like to see improved, feel free to create issues where appropriate to and engage with the broader community via the mailing lists or on issues you create. Welcome to the community and we're looking forward to insights you have regarding the medical world. Adding to our breadth of experiences and backgrounds helps make our project better so we are happy to have your efforts and energies in any way you would like to apply them. Feel free to follow up with any additional questions and we'll be happy to help you along your journey as best we can. --aldrin [1] https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide [2] https://issues.apache.org/jira/projects/MINIFI [3] https://issues.apache.org/jira/projects/MINIFICPP On Wed, Nov 6, 2019 at 9:19 PM Daniel Wang wrote: > Hi NiFi developer team, > > I am a big data admin / developer at medical domain at MD Anderson Cancer > Center. We have been working at data streaming for quite a while. When I do > the use case in medical iot, I can see there are a lot of gaps between > current minifi-nifi interaction, so I would like to join the minifi > developer group and contribute my idea and efforts as a developer. > > I had years of development in Java before becoming a hadoop admin. I had > quite some experiences coding in big data stack, including NiFi processors, > Kafka and Spark. The primary language I am using is Java, but I also use > Python and Scala. > > Anyway, can you please what is the best way to join the developer family? > Looking forward to make a more robust minifi product that is trimmed toward > medical IoT. > > Best, > Daniel Wang > Hadoop Admin and Big data developer > MD Anderson Cancer Center
