Start flow automatically on Nifi start

Hello, I have a flow combining several independent processes (different data sources, different data targets). I am looking for answers on the following questions, so far nothing on the doc. Maybe some of the experts here can help: - I would like to start some of the flows whenever the nifi serve

Re: Start flow automatically on Nifi start

- I would like to start some of the flows whenever the nifi server is restarted. I think I can use 'nifi.flowcontroller.autoResumeState' to achieve that. Or would that start all processors? Yes, that's the purpose of this property and it's enabled by default. Did you try it and not see expect

Re: Nifi Clustering - work distribution on workers

Hi Mark: Thanks for your answers but being a newbie I am still not clear about some issues: Regarding hdfs multiple files: Typically, if you want to pull from HDFS and partition that dataacross the cluster, you would run ListHDFS on the Primary Node only, and then use Site-to-Site [1] to distrib

Re: Start flow automatically on Nifi start

David, To expand on the second part of Andrew's answer, you can make templates of the Process Groups you divide your flow into so that you can add only the ones you need to any given cluster. Brandon On Thu, Oct 15, 2015 at 12:16 PM, Andrew Grande wrote: > > > - I would like to start some of t

RE: Start flow automatically on Nifi start

Thanks for the answers! So I guess I will need to create the flow programatically using Nifi REST API depending on the specifics of the cluster being setup, essentially: Create processor group -> instantiate template Will give that a try. Long term, I thing Nifi could accept several flows definit

Re: Start flow automatically on Nifi start

David I think what is worth further clarifying is the fact that NiFi flow lifecycle is based on lifecycle of individual components (e.g., Processors) within the flow. In other words one can start/stop any Processor individually. So, the ‘autoResumeState’ will not start your flow automatically, i

Re: StoreInKiteDataset help

Has anyone gotten Kite to work on HDP? I'd wanted to do this very thing but am running into all kinds of issues with having .jar files not in the distributed cache (basically in /apps/hdp). Any feedback appreciated. -Chris On Sat, Sep 19, 2015 at 11:04 AM, Tyler Hawkes wrote: > Thanks for the

Re: StoreInKiteDataset help

Chris, Are you seeing errors in NiFi or in HDP? If you're seeing errors in NiFi can you please send us the logs? Thanks Joe On Thu, Oct 15, 2015 at 3:02 PM, Christopher Wilson wrote: > Has anyone gotten Kite to work on HDP? I'd wanted to do this very thing but > am running into all kinds of i

Provenance doesn't work with FetchS3Object

I understand that FetchS3Object processor requires an incoming FlowFile to trigger it. The problem is that FetchS3Object emits a RECEIVE provenance event for the existing FlowFile. That event causes following error when I try to open the lineage chart for a simple flow: GenerateFlowFile -> Fetch

Re: Provenance doesn't work with FetchS3Object

Ben I don’t think it needs an incoming FlowFile. It is a scheduled component and will retrieve contents based on how you configure scheduling. Have you tried it without incoming FlowFiles? Cheers Oleg On Oct 15, 2015, at 3:38 PM, Ben Meng mailto:ben.m...@lifelock.com>> wrote: I understand tha

Re: Provenance doesn't work with FetchS3Object

Oleg, Yes, I’ve tried running FetchS3Object without any incoming FlowFile, and it just didn’t generate any output. I’ve also confirmed the behavior by inspecting the codes. The first thing it does is to check if there’s an existing FlowFile, and returns if there isn’t. Regards, Ben From: Ole

Re: Provenance doesn't work with FetchS3Object

Ben, Since FetchS3Object is not creating the FlowFile, it should not be emitting a RECEIVE event. This is certainly a bug. I have created a ticket for this: https://issues.apache.org/jira/browse/NIFI-1038 Thanks -Mark > On Oct 15, 2015, at 3

Re: Provenance doesn't work with FetchS3Object

Thanks Mark. That makes sense. Regards, Ben From: Mark Payne Reply-To: "users@nifi.apache.org" Date: Thursday, October 15, 2015 at 1:09 PM To: "users@nifi.apache.org" Subject: Re: Provenance doesn't work with FetchS3Object Ben, Since F

rest api synchronize

I'm using the REST API to instantiate templates, start process groups, and also remove those process groups. However, when i try to delete a process group through the REST API, I keep getting the following 409 response: "This NiFi instance has been updated by 'anonymous'. Please refresh to synchr

Re: rest api synchronize

Devin, NiFi employs an optimistic locking scheme that requires clients to pass in a revision when making a mutable request. The revision is comprised of a version (a number that increments with each modification) and a client id. The client id can be any string. Typically however, you'll use the c