Hi Koji, The remote process group is configured as follows: URL - http://host02.corp:10000/nifi Transport Protocol - RAW Communications Timeout - 30 sec Yield Duration - 10 sec
Proxy settings are all blank. nifi.remote properties: # Site to Site properties nifi.remote.input.host=host02.corp nifi.remote.input.http.enabled=true nifi.remote.input.http.transaction.ttl=30 sec nifi.remote.input.secure=false nifi.remote.input.socket.port=12000 nifi.remote.input.host and nifi.remote.input.socket.port is different for each node. If the remote process group url doesn't have to point to the ncm anymore, is it recommended to use a vip or dns rr alias? We have 20 instances in this cluster and the remote process group is actually the same cluster so it's redirecting to itself in an attempt to re-distribute work across the cluster from primary node processors. The good news though is that I can't seem to replicate the problem anymore but this was after some revert attempts that was then undone due to the serialization changes (luckily this is in our dev env). I'm going to have to chalk it up to some misconfiguration for now. Thanks, Chien -----Original Message----- From: Koji Kawamura [mailto:ijokaruma...@apache.org] Sent: Sunday, September 11, 2016 11:37 PM To: users@nifi.apache.org Cc: Chien Le <chien...@ds-iq.com>; Kevin Verhoeven <kevin.verhoe...@ds-iq.com>; Wei Zhang <wei.zh...@ds-iq.com>; Ki Kang <ki.k...@ds-iq.com> Subject: Re: Upgrade 0.7.0 to 1.0.0 Hello Kumiko, Sorry to hear that you're having issues with upgrading to 1.0.0. 1) Any NiFi API URL of a node in the remote cluster should work for the target URL. If there're node1, 2, 3 in a remote cluster then "http://node<1, 2 or 3>:<port>/nifi" should work. Every node is capable to handle web requests with NiFi 1.0 Zero Master Clustering. 2) Having 100 outstanding (not finished, under being replicated requests) seems there're some issues. How many Site-to-Site clients do you have? Normally, each client sends request to /nifi-api/site-to-site once per 10 minutes to refresh cached remote site info. Actual data transfer is done with different endpoint. Would you share the Remote Process Group configs such as Transport Protocol and Proxy settings. Also please share thread dump. It can be generated by executing "bin/nifi.sh dump" command, then thread dump is logged to "logs/nifi-bootstrap.log". Thanks, Koji On Sat, Sep 10, 2016 at 7:45 AM, Kumiko Yada <kumiko.y...@ds-iq.com> wrote: > These are two specific questions that we have. > > > > 1) In Nifi Summary/Remove Process Group, what value need to be used for > the Target URI. We were able to use the specific URI in Nifi 0.7.0 because > we had to specified the master node; however, Nifi 1.0.0 uses zero master > clustering and automatically assign the master. > > 2) We are getting the error “2016-09-08 16:00:01,246 ERROR [NiFi Web > Server-6153] o.a.n.c.c.h.r.ThreadPoolRequestReplicator Cannot replicate > request GET /nifi-api/site-to-site because there are 100 outstanding HTTP > Requests already. Request Counts Per URI = {/nifi-api/site-to-site=99, > /nifi-api/flow/current-user=1}. > > In > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/replication/ThreadPoolRequestReplicator.java, > the limit is hardcoded (MAX_CONCURRENT_REQUESTS = 100) in the > ThreadPoolRequestReplicator class, which is completely new with version 1.0. > Are we getting this error because we hit this hard coded value 100? If so, > what is the best workaround, would the increase this value a good idea? > > > > Thanks > > Kumiko > > > > From: Kumiko Yada [mailto:kumiko.y...@ds-iq.com] > Sent: Friday, September 9, 2016 10:06 AM > To: users@nifi.apache.org > Cc: Chien Le <chien...@ds-iq.com>; Kevin Verhoeven > <kevin.verhoe...@ds-iq.com>; Wei Zhang <wei.zh...@ds-iq.com> > Subject: Upgrade 0.7.0 to 1.0.0 > > > > Hello, > > > > We upgraded from 0.7.0 to 1.0.0, and we encountered some issues. We think > that this upgrade brought a change to how the site-to-site calls are made > for the remote process groups and is causing us to hit a max requests limit > that was also introduced with v1.0. Is this a known issue? > > > > The upgrade changes the serialization version of the provenance files that > is not backwards compatibility (nifi-1.0 -> 1-9, nifi-0.7 -> 1-8). Is there > any workaround this? > > > > Thanks > > Kumiko
