Re: Nifi processeror's input form

Just applied the patch and it works flawlessly!! Thanks for the fast response :) Yari On 19 October 2016 at 00:30, Andy LoPresto wrote: > Yari, > > The bug you reported was fixed by Scott and I’ve reviewed and merged it > into master. If you are comfortable checking out the source code and > bu

Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups

Hi, I have finally taken the plunge to upgrade my cluster from 0.6.1 to 1.0.0. 6 nodes with a NCM. With the removal of NCM in 1.0.0 I believe I now have an issue where none of my Remote Process Groups work as they previously did because they were configured to connect to the NCM (as the RPG url)

Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups

Conrad, For s2s now you can just point at any of the nodes in the cluster. Have you tried changing the URL or removing and adding new RPG entries? Thanks Joe On Wed, Oct 19, 2016 at 8:38 AM, Conrad Crampton wrote: > Hi, > > I have finally taken the plunge to upgrade my cluster from 0.6.1 to 1.0

Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups

Regarding the error about the uninheritable flow caused by the "proposed authorizations do not match current authorizations"... that basically means that the node trying to connect has a different set of authorizations (users/groups/policies) from what the other nodes in the cluster have. This most

Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups

Hi Joe, Yep, Tried removing the RPG that referenced the NCM and adding new one with one of the datanodes as url. That sort of worked, but kept getting errors about the NCM not being available for the ports and therefore couldn’t actually enable the port I needed to for that RPG. Thanks Conrad

Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups

Hi Joe, Yep, Tried removing the RPG that referenced the NCM and adding new one with one of the datanodes as url. That sort of worked, but kept getting errors about the NCM not being available for the ports and therefore couldn’t actually enable the port I needed to for that RPG.

Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups

Hi, Ok, so I have now connected my (old NCM) server to the cluster, but now I am getting reports in the other server logs that can’t connect due to SSL handshake exception… 2016-10-19 14:57:02,363 WARN [NiFi Site-to-Site Connection Pool Maintenance] o.apache.nifi.remote.client.PeerSelector org

Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups

thanks conrad - did get it. Bryan is being more helpful that I so I went silent :-) On Wed, Oct 19, 2016 at 10:02 AM, Conrad Crampton wrote: > Hi Joe, > Yep, > Tried removing the RPG that referenced the NCM and adding new one with > one of the datanodes as url. > That sort of worked

Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups

Ok that does seem like a TLS/SSL issue... Is this a single cluster doing site-to-site to itself? On Wed, Oct 19, 2016 at 10:06 AM, Joe Witt wrote: > thanks conrad - did get it. Bryan is being more helpful that I so I > went silent :-) > > On Wed, Oct 19, 2016 at 10:02 AM, Conrad Crampton > wr

Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups

Yes. All 6 nodes and 1 NCM were the original set up After upgrade not fussed whether I lost the NCM and went to just the 6 nodes or re-introduced the redundant NCM as a new node (for actually doing stuff). I had to go the latter route in the end as the RPG were complaining about it not being the

Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups

One other thing… The RPGs have an unlocked padlock on them saying S2S is not secure. Conrad From: Bryan Bende Reply-To: "users@nifi.apache.org" Date: Wednesday, 19 October 2016 at 15:20 To: "users@nifi.apache.org" Subject: Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups Ok that

Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups

Trying to think of things to check here... Does every node have nifi.remote.input.secure=true in nifi.properties and the URL in the RPG is an https URL? On Wed, Oct 19, 2016 at 10:25 AM, Conrad Crampton < conrad.cramp...@secdata.com> wrote: > One other thing… > > The RPGs have an unlocked padloc

Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups

Hi, Yes, every nifi.properties is set thus – with host and port different for each. # Site to Site properties nifi.remote.input.socket.host=ncm.xxx nifi.remote.input.socket.port=9870 nifi.remote.input.secure=true nifi.remote.input.http.enabled=false nifi.remote.input.http.transaction.ttl=30 se

Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups

Yes http site-to-site was added recently so setting that to disabled should be fine and not related. If you are using all the same keystores and truststores from before, then I can't think of why the nodes wouldn't be able to communicate securely. Unless anyone else has some other ideas, you may

Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups

Hi, The nifi-user.log doesn’t show any errors – in fact it shows success for any authentication to the old NCM server. What is odd though is the old NCM server is the only one out of the 7 servers that I can’t log into at https://:9090/nifi where I can with all the others on their respective

Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups

Hi, As a plan for tomorrow – I have generated new keystores, truststores, client certts etc. for all nodes in my cluster using the From: Bryan Bende Reply-To: "users@nifi.apache.org" Date: Wednesday, 19 October 2016 at 15:33 To: "users@nifi.apache.org" Subject: Re: Upgrade 0.6.1 to 1.0.0 prob

RE: How to increase the processing speed of the ExtractText and ReplaceText Processor?

I’m not clear on how much data you are processing, does the data(.dat) file have 3,00,000 rows? Kevin From: prabhu Mahendran [mailto:prabhuu161...@gmail.com] Sent: Wednesday, October 19, 2016 2:05 AM To: users@nifi.apache.org Subject: Re: How to increase the processing speed of the ExtractText a

Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups

That is definitely weird that it is only an issue on the node that used to be the NCM. Might be worth double checking the keystore and truststore of that one node and make sure it has what you would expect, and also double check nifi.properties compared to the others to see if anything seems differ

Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups

Hi Conrad, Bryan is correct that changing the certificates (and the encapsulating keystores and truststores) will not affect any data held in the nodes. Regenerating everything using the TLS toolkit should hopefully not be too challenging, but I am also curious as to why you are getting these h

Re: Upgrade 0.6.1 to 1.0.0 problems with Remote Process Groups

Also worth noting that there were some changes to how cluster and site-to-site communications worked with certificates between 0.6.1 and 1.0.0: * https://issues.apache.org/jira/browse/NIFI-1981 (0.6.1 -> 0.7.0) * https://issues.apache.org/jira/br

Re: How to increase the processing speed of the ExtractText and ReplaceText Processor?

Prabu, In order to move 3M rows in 10 minutes, you'll need to process 5000 rows/second. During your 4 hour run, you were processing ~200 rows/second. Without any new optimizations you'll need ~25 threads and sufficient memory to feed the threads. I agree with Mark and you should be able to get f

ApacheCon is now less than a month away!

Dear Apache Enthusiast, ApacheCon Sevilla is now less than a month out, and we need your help getting the word out. Please tell your colleagues, your friends, and members of related technical communities, about this event. Rates go up November 3rd, so register today! ApacheCon, and Apache Big Dat

Re: Calculating the theoretical throughput of a Nifi server

I've noticed that the total# of flow files and processors is missing from the questions. Since NIFI keeps state on disk, every transaction has to be committed. Some newer processors support batch mode, but worst case is absent of this. Thus the limit might not be in bytes but number of flow file