chad there have been a ton of improvements overall. but the causes of disconnects are much better handled and largely resolved. for cases that remain due to general distrubuted systems fun auto join is better.
so def happy to hear your findings with 1.8 thanks On Mon, Dec 10, 2018, 10:48 AM Woodhead, Chad <chad.woodh...@ncr.com wrote: > Adding my response to Joe since I responded to the wrong thread: > > > > Joe, > > > > Our current dev/cert/prod flows run on NiFi 1.2.0 and this is where we > face this issue a lot (hence why we created the first version of the > monitoring flow). We have stood up new clusters which are running NiFi > 1.8.0. No users/flows are running on the new 1.8.0 clusters besides our > Admin monitoring flows. So no we don’t face this issue on 1.8.0, but we > also have no users yet. We are opening up the 1.8.0 clusters to our users > this week for them to start porting over their existing flows to the 1.8.0 > clusters. > > > > Jeremy, > > > > I have a reverse proxy setup and can use it successfully for the NiFi UI. > But since the disconnected node is technically still up and running, I > still get sent to the disconnected node. So the API call to the reverse > proxy will still sometimes give me the “Only a node connected to a cluster > can process the request.”” message since it will get sent to the > disconnected node sometimes. > > > > But it sounds like I might be going down a path that is not worth the > effort, since you guys are saying 1.8.0 shouldn’t give me these issues > anymore. I can hold off on future development of this and reach back out if > I do face the disconnecting issue in 1.8.0 as well. > > > > For my knowledge, is it something in 1.8.0 specifically that > addresses/fixes this problem, or just overall improvements since 1.2.0? > > > > Thanks, > > Chad > > > > > > *From: *Jeremy Dyer <jdy...@gmail.com> > *Reply-To: *"users@nifi.apache.org" <users@nifi.apache.org> > *Date: *Monday, December 10, 2018 at 10:26 AM > *To: *"users@nifi.apache.org" <users@nifi.apache.org> > *Subject: *Re: How to get Node ID of disconnected node to be used in a > cluster/node monitoring flow > > > > Chad - Ahh I see what you are saying now. To Joe's point the problems > really shouldn't be there anymore. However to resolve what you are seeing > the best practice is to setup the cluster behind a reverse proxy [1]. In > this configuration you are not accessing the REST API on an explicit node > and the issue you explained will not occur. > > > > [1] - > https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#proxy_configuration > <https://urldefense.proofpoint.com/v2/url?u=https-3A__nifi.apache.org_docs_nifi-2Ddocs_html_administration-2Dguide.html-23proxy-5Fconfiguration&d=DwMFaQ&c=gJN2jf8AyP5Q6Np0yWY19w&r=MJ04HXP0mOz9-J4odYRNRx3ln4A_OnHTjJvmsZOEG64&m=Fu5GvEY5W6MSQUzs1kOAlkIcfLFh4ZZCXJPyGx-7AKQ&s=E4jDUra2uIvmsWV4bPBLS5unz_Jl6yUgPvD-ShNcLMI&e=> > > > > On Mon, Dec 10, 2018 at 10:23 AM Joe Witt <joe.w...@gmail.com> wrote: > > chad > > > > in 1.8.0 you should not see much need to script that. are you still > seeing disconnects in 1.8.0 that it doesnt restore on its own. > > > > thanks > > > > On Mon, Dec 10, 2018, 10:08 AM Woodhead, Chad <chad.woodh...@ncr.com > wrote: > > Hey Jeremy! > > > > We occasionally see a node disconnect from the cluster (not disappear from > the cluster) so I’m building this flow just to reconnect a disconnected > node. The cluster still shows the node, but just that it is in a > disconnected state. > > > > As far as the API call to /nifi-api/controller/cluster, yes I get the same > result if I call the API on one of the connected nodes. If I call the API > of a disconnected node, I get “Only a node connected to a cluster can > process the request.” And since the flowfile is sometimes run and processed > on the disconnected node, using ${hostname(true)} in the API to > /nifi-api/controller/cluster won’t work. > > > > I am also using NiFi 1.8.0 for reference as well. > > > > -Chad > > > > *From: *Jeremy Dyer <jdy...@gmail.com> > *Reply-To: *"users@nifi.apache.org" <users@nifi.apache.org> > *Date: *Monday, December 10, 2018 at 9:12 AM > *To: *"users@nifi.apache.org" <users@nifi.apache.org> > *Subject: *Re: How to get Node ID of disconnected node to be used in a > cluster/node monitoring flow > > > > **External Message* - Use caution before opening links or attachments* > > > > Chad - The API call to /nifi-api/controller/cluster will return all of the > nodes even if they are disconnected. The JSON response will indicate which > node is "disconnected". I have included a screenshot of a simple > environment I have setup for this where I just tested it. However I'm > thinking since you are performing monitoring here you might have a deeper > issue you are trying to work to monitor for. Are you experiencing > situations where you are seeing nodes completely disappear from the cluster > and not just be in the "disconnected" state? If so that is a different > issue but if they are just "disconnected" you should be able to see that as > the screenshot below illustrates. I am using NiFi 1.8.0 just for reference. > > > > On Mon, Dec 10, 2018 at 8:47 AM Woodhead, Chad <chad.woodh...@ncr.com> > wrote: > > I am working on a cluster/node monitoring flow and was looking for some > guidance. > > > > I currently have a working flow that monitors if a node gets disconnected > from the cluster and sends an email if a node disconnects. > > > > Here is the flow: > > (A SiteToSiteBulletinReportingTask sends to the ClusterMonitoring input > port) > > *Error! Filename not specified.* > > > > The CheckAPIForDisconnectedNode process group makes an API call to NiFi > confirm the node is still disconnected before sending the email. > > > > This flow works as expected. I am now wanting to update the flow to > automatically attempt to reconnect the node itself. When a node > disconnects, I capture the server name of that node. To reconnect a node, > you can call the API with PUT /nifi-api/controller/cluster/nodes/{id}, but > the problem is I don’t know the Node ID. The only place I have found I can > get the Node ID is via a GET /nifi-api/controller/cluster, but that API > only works on a node that is connected to the cluster and the way this flow > works, the processing can and sometimes does actually run on the node that > has been disconnected. Is there a way to force it so the processing is only > done on a cluster connected node and then use ${hostname(true)} in the API > call, or is there another way I can get the Node ID, even on a disconnected > node? I don’t want to have to hard code all of my Node ID’s somewhere. > > > > Thanks, > > Chad Woodhead > > > > > > > >
