Re: [EXT] Re: Primary Only Content Migration

There is an existing JIRA submitted by Pierre. I think its goal is the same with what Joe mentioned above. https://issues.apache.org/jira/browse/NIFI-4026 As for hashing and routing data with affinity/correlation, I think 'Consistent Hashing' is the most popular approach to minimize the impact of

Re: [EXT] Re: Primary Only Content Migration

Peter I'm not sure there is a good way for a processor to drive such a thing with existing infrastructure. The processor having ability to know about the structure of a cluster is not something we have wanted to expose for good reasons. There would likely need to be a more fundamental point of

RE: [EXT] Re: Primary Only Content Migration

Joe, I agree it is a lot of work, which is why I was thinking of starting with a processor that could do some of these operations before looking further. If the processor could move flowfile's between nodes in the cluster it would be a good step. Data comes in form a queue on any node, but

Re: [EXT] Re: Primary Only Content Migration

Peter, It isn't a pattern that is well supported now in a cluster context. What is needed are automatically load balanced connections with partitioning. This would mean a user could select a given relationship and indicate that data should automatically distributed and they should be able to

RE: [EXT] Re: Primary Only Content Migration

Bryan, We see this with large files that we have split up into smaller files and distributed across the cluster using site-to-site. We then want to merge them back together, so we send them to the primary node before continuing processing. --Peter -Original Message- From: Bryan Bende

Re: Primary Only Content Migration

Peter, There really shouldn't be any non-source processors scheduled for primary node only. We may even want to consider preventing that option when the processor has an incoming connection to avoid creating any confusion. As long as you set source processors to primary node only then everything

Primary Only Content Migration

I'm sure many of you have the same situation, a flow that runs on a cluster, and at some point merges back down to a primary only processor; your files sit there in the queue with nowhere to go... We've used the work around of having a remote processor group that loops the data back to the

Re: [DISCUSS] Change of Cluster Flow Inheritance

Otto, I am not currently proposing that there be a unified flow comparison capability - or that there not be one. I think that's more of an implementation detail. I do believe there's a good chance that we can map a FlowSnippetDTO into a VersionedProcessGroup object, in which case we could use

Re: [DISCUSS] Change of Cluster Flow Inheritance

Thanks, all, for the great feedback so far. I did leave out any mention of how I would envision handling the authorizations.xml and users.xml in the new approach, but yes, as has been suggested I do believe that it will be important to also inherit those from the cluster. Bryan also brought up

Re: Java 10

All, There is work on-going to allow NiFi to: A) Run on Java 9, 10, and beyond. B) Compile on Java 9, 10, and beyond. C) Code leveraging Java 9, 10, and beyond capabilities. We can make things in category A happen anytime and these should be top priority. We can make things happen in category B

Re: Java 10

I would say that's a safe assumption. Given the newly aggressive release cadence of Java, I would also caution you to be careful because a lot of the transitive dependencies might suddenly break between releases. On Thu, Jun 7, 2018 at 10:38 AM Jon Logan wrote: > Will any PRs with Java 9

Re: [DISCUSS] Change of Cluster Flow Inheritance

Mark I definitely think it is time to move on from the fingerprinting model. I recall the conversations long ago that led us down this path and ultimately the thing that mattered most was ensuring data loss cases were prevented. This still addresses that, reduces a ton of code, and simplifies

Re: [DISCUSS] Change of Cluster Flow Inheritance

Great ideas Mark. Another addition for ease/convenience of a node joining a cluster is related to the authorizations.xml and users.xml. In the case of the flow.xml.gz, if the file is missing, it will obtain a copy from the cluster. The same should be true of authorizations.xml and users.xml files,

Re: Java 10

Will any PRs with Java 9 features be rejected until then? (ie. preserving Java-8 compatibility) Is there a plan for supporting multiple Java versions? On Thu, Jun 7, 2018 at 10:18 AM, Mike Thomsen wrote: > Once that is done, it should run on Java 10. However, under the new Oracle > version

Re: Question on Docker module

Hi, Disclaimer: I've been working with the nifi-docker module for only a couple of weeks now so I'm relatively new to this. Unfortunately there's no way I'm aware of right now using a pure Dockerfile to implement a build time switch for cases like this where you have a a) locally built binary b)

Re: [DISCUSS] Change of Cluster Flow Inheritance

Using the versioned flow logic seems like a good idea. Would the authorizer fingerprints still be checked as part of joining the cluster? Currently that is appended to the overall fingerprint to ensure each node has the same users/policies, or at least same config (i.e. LDAP). Would be nice if a

Re: Java 10

Once that is done, it should run on Java 10. However, under the new Oracle version rules Java 10 is a short term iteration. Java 11 will probably be the next version officially supported by NiFi because it's supposed to be Java's next LTS release. Java 10 features like the var key will also be

Re: [DISCUSS] Change of Cluster Flow Inheritance

Great write up. While I am not an expert on clustering, it would seem that having one method of comparing flows, perhaps with different strategies within that would be more maintainable as well. Are you proposing that there is a unified flow comparison capability/implementation/service that is

Re: [DISCUSS] Change of Cluster Flow Inheritance

Hi Mark, Thanks for the detailed explanations! The suggested approach makes total sense to me and it'll greatly improve the user experience as I often see a node unable to join the cluster because of a discrepancy in some component configuration... Right now, the only solution is to do as you

Re: [DISCUSS] Change of Cluster Flow Inheritance

Hi Mark, Thanks for the great write-up! I support your proposal. It seems a logical improvement based on your description of how fingerprinting works today, the original problem it was trying to solve, and the proposed alternative. I am not an expert on how NiFi clustering is implemented, so

Re: Java 10

You can track progress here: https://issues.apache.org/jira/browse/NIFI-5174 There has been some progress very recently. On June 7, 2018 at 09:40:50, Sivaprasanna (sivaprasanna...@gmail.com) wrote: Nope. Not yet. AFAIK, there is a Jira to support Java 9. On Thu, Jun 7, 2018 at 7:09 PM,

[DISCUSS] Change of Cluster Flow Inheritance

Hi all, Over the past couple of months, I have been doing a lot of testing with large scale flows and talking to others who are using large scale flows in production. ("Large scale" flows in this case means several thousand to tens of thousands of Processors). While NiFi does a really good job

Re: nifi not running (installation problem) - brew

Hello, I suspect the issue is your Java version. We are in the process of providing support for those beyond 8. In the interim, you can get it working via the following: brew tap caskroom/versions > brew update (if it doesn't do it automatically) > brew cask install java8 I'll look to see

Re: nifi not running (installation problem) - brew

To Kevin's point Java 10 or even 9 for that matter is not yet supported. If you don't want to use mess with your existing environment setup we do offer a docker image[1] that you can run if interested. [1] - https://hub.docker.com/r/apache/nifi/ Thanks, Jeremy Dyer On Thu, Jun 7, 2018 at 9:41

Re: nifi not running (installation problem) - brew

The version of NiFi you are using (1.6.0) requires Java 8 (jre or jdk 1.8). Regards, Kevin On 6/7/18, 09:38, "kirilzilla" wrote: Hi, im writing my bachelor thesis and would like to use apache nifi. im using macOS 10.13.5 right now and installed nifi over brew packet manager "brew

Re: Java 10

Nope. Not yet. AFAIK, there is a Jira to support Java 9. On Thu, Jun 7, 2018 at 7:09 PM, kirilzilla wrote: > does nifi support java 10 ? or even java 9 ? > > > > -- > Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/ >

Re: Java 10

does nifi support java 10 ? or even java 9 ? -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

nifi not running (installation problem) - brew

Hi, im writing my bachelor thesis and would like to use apache nifi. im using macOS 10.13.5 right now and installed nifi over brew packet manager "brew install nifi". i can't access the user interface, because nifi doesn't seem to start. http://localhost:8080/nifi if i run "nifi start" this is

Re: Question on Docker module

Sivaprasanna - They are very similar but different. Exists to primarily assist with speeding up a developers dev cycle. Its purpose is to allow you to make source code changes to NiFi, build with maven (including docker image), and then run the container with your latest changes. So in a sense you

Question on Docker module

Team, In the project's nifi-docker module, we have two things: dockerhub and dockermaven. Are they one and the same or they are there for different purposes? I have to warn you I don't know much about 'dockerfile-maven-plugin' so some light

Re: API endpoint for /resources

I'm not sure it is in the UI, but you should be able to use the REST API to create a policy and set the resource to "/resources". I don't have an example handy, but if you looked in Chrome Dev tools while creating another policy you could see what the UI normally submits and just modify it. Might

Re: API endpoint for /resources

Bryan, How does one get READ for /resources? Put another way, what policy corresponds to /resources? The user has been added to all policies available through the UI, and /resources does not appear in authorizations.xml. Thanks, -Mark On Thu, Jun 7, 2018 at 8:37 AM, Bryan Bende wrote: > Mark,

Re: [ANNOUNCE] New NiFi PMC member Mike Thomsen

Thanks! On Thu, Jun 7, 2018 at 7:55 AM Mike Hogue wrote: > Congratulations! > On Thu, Jun 7, 2018 at 02:31 Jeff wrote: > > > Congrats, Mike! > > > > On Wed, Jun 6, 2018 at 10:52 PM Michael Moser > wrote: > > > > > Congrats and a big thank you for your work! > > > > > > > > > On Wed, Jun 6,

API endpoint for /resources

What information specifically does the Rest API endpoint "/resources" provide? Also, a user belongs to every policy, and when hitting that endpoint still receives "No applicable policies could be found. Contact the system administrator." Is the "/resources" endpoint still supported? Thanks, Mark

Re: [ANNOUNCE] New NiFi PMC member Mike Thomsen

Congratulations! On Thu, Jun 7, 2018 at 02:31 Jeff wrote: > Congrats, Mike! > > On Wed, Jun 6, 2018 at 10:52 PM Michael Moser wrote: > > > Congrats and a big thank you for your work! > > > > > > On Wed, Jun 6, 2018 at 10:10 PM, Scott Aslan > > wrote: > > > > > Congrats > > > > > > On Wed, Jun

Re: [ANNOUNCE] New Apache NiFi Committer Sivaprasanna Sethuraman

Congratulations! On Thu, Jun 7, 2018 at 00:50 Sivaprasanna wrote: > Thanks everyone :) > > On Thu, Jun 7, 2018 at 2:01 AM, Jorge Machado wrote: > > > Congrats ! > > > > Jorge > > > > > > > > > > > > > On 6 Jun 2018, at 18:44, Otto Fowler wrote: > > > > > > Congratulations! > > > > > > > > > On

Re: Issue securing nifi with ca-server.

Hey Henk! Could you make any progress remote debugging your issue? Although not guaranteed to help, you may try and use strace to look into what's happening on the syscall level. For example I've ran this command after starting the server: strace -fp -s 100 2>&1 | grep -v clock_gettime | grep

Re: [ANNOUNCE] New NiFi PMC member Mike Thomsen

Congrats, Mike! On Wed, Jun 6, 2018 at 10:52 PM Michael Moser wrote: > Congrats and a big thank you for your work! > > > On Wed, Jun 6, 2018 at 10:10 PM, Scott Aslan > wrote: > > > Congrats > > > > On Wed, Jun 6, 2018, 9:03 PM Matt Burgess wrote: > > > > > Congratulations Mike! Great to have