Re: Queue incoherent state

2017-05-16 Thread Arnaud G
Hi Matt, Thanks for your reply! I finally solved the problem by deleting all the content in the flowfile directory, but here are my observations: 1) The problem was coming from one of the cluster node, when this node was out of the cluster, the queue were reporting 0 flowfile. 2) The first time

Re: Queue incoherent state

2017-05-16 Thread Matt Gilman
Thanks for following up with these details. I'm going to add these observations to the corresponding JIRA [1]. Thanks and sorry again for the inconvenience. Matt [1] https://issues.apache.org/jira/browse/NIFI-3897 On Tue, May 16, 2017 at 3:34 AM, Arnaud G wrote: > Hi Matt, > > Thanks for your

Re: Queue incoherent state

2017-05-16 Thread Mark Payne
Arnaud, Did you have any WARN or ERROR messages in the logs? I'm particular interested in anything that mentions the word "Swap" or "swap" (i.e., regardless of case). Is it possible that the FlowFile Repository could have run out of disk space? Thanks -Mark On May 16, 2017, at 3:34 AM, Arnaud

Re: Queue incoherent state

2017-05-16 Thread Arnaud G
Hi again, Unfortunately I didn't save the logs when the node restarted, but I don't remember anything that provided me a clue regarding the reason of the blocked queue. I just have a few logs during the week-end when the queues were in this strange state: 2017-05-14 09:01:29,635 INFO [pool-12-th

Service controller list NiFi 0.x to 1.x

2017-05-16 Thread Russell Bateman
It appears to me that that, unlike what happened in NiFi 0.x, in 1.x when I look at controller services via the General menu -> Controller Services, what I see is totally different from what I see when I configure controller services for a processor. If I use the General menu to set up my cont

Re: Service controller list NiFi 0.x to 1.x

2017-05-16 Thread Andrew Lim
Hi Russell, Thanks for your question. Yes, working with Controller Services has definitely changed in 1.x compared to 0.x NiFi. Matt Gilman wrote a nice article about how Controller Service scoping was updated in 1.x with the introduction of Multi-Tenant Authorization and also discusses the r

Re: How to lock getfile upto putfile write into same file?

2017-05-16 Thread Juan Sequeiros
Prabhu, PutFile should pre-pend files with a "." while it is writing to it and then in the end will copy the "." file to its "filename" value. On GetFile side make sure that "ignore hidden files" is set to true. On Tue, May 16, 2017 at 1:29 AM prabhu Mahendran wrote: > I have scheduled getfi

Re: Reliable operation use case

2017-05-16 Thread Andy Yar
Thank your for the info. I guess I will try a small PoC and see how it goes. On Mon, May 15, 2017 at 8:13 PM, Pierre Villard wrote: > Hi Andy, > > Regarding bulletins, you can use the new SiteToSiteBulletinReportingTask [1] > to achieve what you are expecting (in NiFi 1.2.0). For provenance, you

Re: Service controller list NiFi 0.x to 1.x

2017-05-16 Thread Russell Bateman
Thanks, Andrew! (From [1]:) "/This means that the service will be available to all Processors and Controller Services defined in that Process Group _and below_./" In my experience, this isn't true. If I create a controller _via the General menu_ in the very root of my NiFi canvas, configure

Re: Service controller list NiFi 0.x to 1.x

2017-05-16 Thread Michael Moser
Hi Russ, In NiFi 1.2.0, there are 3 Reporting Tasks (SiteToSiteBulletionReportingTask, SiteToSiteProvenanceReportingTask, and SiteToSiteStatusReportingTask) that have the capability of using a Controller Service (StandardSSLContextService). So yes, the Global -> Controller Settings has a limited

Re: Service controller list NiFi 0.x to 1.x

2017-05-16 Thread Russell Bateman
We're still testing our migration from 0.7.2 to 1.1.2 which is imminent. We still don't know how much of this is going to have to be reconfigured. Our down-streamers tend to be on the grouchy side, so we're testing to see what's going to irritate them. Were it not for the magic ids, I'd feel sa

Re: Service controller list NiFi 0.x to 1.x

2017-05-16 Thread Andy LoPresto
Russell, Is there a reason you are planning your migration to 1.1.2 and not 1.2.0? There are a number of significant feature and bug fixes that went into 1.2.0 that will likely make your experience much better. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC

Re: Service controller list NiFi 0.x to 1.x

2017-05-16 Thread Russell Bateman
It's because product planning can't be done overnight. It was a fight (months ago) to get higher approval to move to1.x. 1.1.2was the current release when approval was given. We'll plan to move ahead to 1.2.xin a few months motivated by the next release. In development, I tend to keep up, but p

variable url for remote process group

2017-05-16 Thread mohammed shambakey
Hi Is it possible to use variables for remote process group, as the remote server IP is not known in advance? If not, the ONLY alternative for remote communication will be HTTP processors (e.g., handlehttprequest/response) instead of site-to-site, right? Regards -- Mohammed

Re: Service controller list NiFi 0.x to 1.x

2017-05-16 Thread Russell Bateman
As promised I'm posting back that all of our controller services are migrated from any flow.xml.gx created by 0.7.2 and put into the top-level process group so that we have no problem accessing them. Our biggest problem is then education on how best to develop flows with controller services beg