Re: setting processor concurrency based on the development/production environment

Boris, Here's an idea youncould explore _today_. Assume your dev and prod flows live in different bucket/registry instance. Given that you are trying out NiFi 1.6, you should be able to extract the versioned flow from DEV and process it to change the concurrency level for PROD before committing

Re: Nested Compressed Files

That is exactly what I needed. I didn’t realize back pressure could be disabled. Thanks Shawn Sent from my iPhone On Mar 1, 2018, at 7:23 AM, Brandon DeVries > wrote: Shawn, Often when you have a loop you can run into problems with back pressure. I'd try

List archive files before extration

Hello, I've got a NiFi flow which: 1. Ingest archive files (tar.gz, rar and zip) 2. IdentifyMimeType of the archive 3. UnpackContent of the archive 4. Identify which of the files can be processed based on filename The problem I've got is that a lot of processing time/content repo space is

Re: setting processor concurrency based on the development/production environment

I think you could put it under either project. Ultimately, if we go with that approach, most (all?) of the logic/enhancement would be in the NiFi code base during save version / import flow / change version operations, so probably best to create it there. Glad you are finding NiFi useful.

Re: setting processor concurrency based on the development/production environment

thanks Bryan and Kevin. I will be happy to open a jira - would it be a NiFi jira or NiFi registry? I like the approach that Bryan suggested. I guess for now I will just color code the processors that need to be changed in production. P.S. I really, really like where NiFi is going...I've looked

Re: setting processor concurrency based on the development/production environment

Hi Boris, Good point regarding concurrent tasks; thanks for sharing! This is a great candidate for something that one should be able to create environment-specific values for, as Bryan suggests. I agree we should create a NiFi JIRA to track this enhancement. Thanks, Kevin On 3/1/18, 11:44,

Re: setting processor concurrency based on the development/production environment

Hello, Glad you are having success with NiFi + NiFi Registry! You brought up an interesting point about the concurrent tasks... I think we may want to consider making the concurrent tasks work similar to variables, in that we capture the concurrent tasks that the flow was developed with and

setting processor concurrency based on the development/production environment

Hello NiFi community, started using NiFi recently and fell in love with it! We run 1.6 NiFi alone with new NiFi registry and I am trying to figure out how to promote NiFi flow, created in VM environment to our cluster. One of the things is "Concurrent Tasks" processor parameter. I bump it to 2

Re: Atlas and NiFi integration help

Mike, That is basically the point of SiteToSiteProvenanceReportingTask... you send the provenance events from reporting task back to the same cluster, and then leverage existing processors like the ElasticSearch processors. Otherwise we'd get into building 100 reporting tasks for all the various

Re: Extract primary node hostname/ipaddress in the flow

Hi, There is the following JIRA: https://issues.apache.org/jira/browse/NIFI-1598 Pierre 2018-03-01 15:53 GMT+01:00 Bryan Bende : > Hello, > > There is a hostname() function in EL, but there is not one > specifically for the primary node. > > I think this has come up before

Re: Atlas and NiFi integration help

Bryan, I have a feeling you're right. This might call for a reporting task that exports to ElasticSearch so that Kibana dashboards can be used to answer these questions. Thanks, Mike On Thu, Mar 1, 2018 at 10:20 AM, Bryan Bende wrote: > Mike, > > As far as I know, Atlas is

Re: Atlas and NiFi integration help

Mike, As far as I know, Atlas is not really about "event level" lineage, it is more about "system level" or "data set' level. So I believe the goal of Atlas is to show how the systems are connected and how a particular data set flows through the system. So an example might be... NiFi pulls from

Re: Extract primary node hostname/ipaddress in the flow

Hello, There is a hostname() function in EL, but there is not one specifically for the primary node. I think this has come up before and probably makes sense, it just hasn't been implemented yet. -Bryan On Thu, Mar 1, 2018 at 9:28 AM, ashmeet kandhari wrote: > Hi

Extract primary node hostname/ipaddress in the flow

Hi all, Is there a way in nifi flows to get the primary node hostname or ip in variable form eg: ${primary_node_hostname}. UseCase: ListenHTTP processor running on the primary node, then I want to set the URL dynamically like http://${primary_node_hostname}/endPoint. Thanks, Ashmeet Kandhari

Re: Nested Compressed Files

Shawn, Often when you have a loop you can run into problems with back pressure. I'd try setting the back pressure limits to 0 / 0 B on one or more of the relationships in the loop so the involved processors don't get "stuck". Brandon On Wed, Feb 28, 2018 at 12:26 PM Shawn Weeks