That's fair.  But for the sake of total clarity on my own part, after one
of these disaster scenarios with a newly quorum-elected primary things
cannot be driven through the UI and at least through parts the REST API.

I just ran through the following.  We have 3 nodes A, B, C with A primary,
and A becomes unreachable without first disconnecting.  Then B and C may (I
haven't verified) continue operating the flow they had in the clusters'
last "good" state.  But they do elect a new primary, as per the REST
nifi-api/controller/cluster response.  But now the flow can't be changed,
and in some cases it can't be reported on either, i.e. some GETs fail, like
nifi-api/flow/process-groups/root.

Are we describing the same behavior?

On Fri, May 19, 2017 at 11:12 AM, Joe Witt <joe.w...@gmail.com> wrote:

> If there is no longer a quorum then we cannot drive things from the UI
> but the cluster remaining is in tact from a functioning point of view
> other than being able to assign a primary to handle the one-off items.
>
> On Fri, May 19, 2017 at 11:04 AM, Neil Derraugh
> <neil.derra...@intellifylearning.com> wrote:
> > Hi Joe,
> >
> > Maybe I'm missing something, but if the primary node suffers a network
> > partition or container/vm/machine loss or becomes otherwise unreachable
> then
> > the cluster is unusable, at least from the UI.
> >
> > If that's not so please correct me.
> >
> > Thanks,
> > Neil
> >
> > On Thu, May 18, 2017 at 9:56 PM, Joe Witt <joe.w...@gmail.com> wrote:
> >>
> >> Neil,
> >>
> >> Want to make sure I understand what you're saying.  What are stating
> >> is a single point of failure?
> >>
> >> Thanks
> >> Joe
> >>
> >> On Thu, May 18, 2017 at 5:27 PM, Neil Derraugh
> >> <neil.derra...@intellifylearning.com> wrote:
> >> > Thanks for the insight Matt.
> >> >
> >> > It's a disaster recovery issue.  It's not something I plan on doing on
> >> > purpose.  It seems it is a single point of failure unfortunately.  I
> can
> >> > see
> >> > no other way to resolve the issue other than to blow everything away
> and
> >> > start a new cluster.
> >> >
> >> > On Thu, May 18, 2017 at 2:49 PM, Matt Gilman <matt.c.gil...@gmail.com
> >
> >> > wrote:
> >> >>
> >> >> Neil,
> >> >>
> >> >> Disconnecting a node prior to removal is the correct process. It
> >> >> appears
> >> >> that the check was lost going from 0.x to 1.x. Folks reported this
> JIRA
> >> >> [1]
> >> >> indicating that deleting a connected node did not work. This process
> >> >> does
> >> >> not work because the node needs to be disconnected first. The JIRA
> was
> >> >> addressed by restoring the check that a node is disconnected prior to
> >> >> deletion.
> >> >>
> >> >> Hopefully the JIRA I filed earlier today [2] will address the phantom
> >> >> node
> >> >> you were seeing. Until then, can you update your workaround to
> >> >> disconnect
> >> >> the node in question prior to deletion?
> >> >>
> >> >> Thanks
> >> >>
> >> >> Matt
> >> >>
> >> >> [1] https://issues.apache.org/jira/browse/NIFI-3295
> >> >> [2] https://issues.apache.org/jira/browse/NIFI-3933
> >> >>
> >> >> On Thu, May 18, 2017 at 12:29 PM, Neil Derraugh
> >> >> <neil.derra...@intellifylearning.com> wrote:
> >> >>>
> >> >>> Pretty sure this is the problem I was describing in the "Phantom
> Node"
> >> >>> thread recently.
> >> >>>
> >> >>> If I kill non-primary nodes the cluster remains healthy despite the
> >> >>> lost
> >> >>> nodes.  The terminated nodes end up with a DISCONNECTED status.
> >> >>>
> >> >>> If I kill the primary it winds up with a CONNECTED status, but a new
> >> >>> primary/cluster coordinator gets elected too.
> >> >>>
> >> >>> Additionally it seems in 1.2.0 that the REST API no longer support
> >> >>> deleting a node in a CONNECTED state (Cannot remove Node with ID
> >> >>> 1780fde7-c2f4-469c-9884-fe843eac5b73 because it is not
> disconnected,
> >> >>> current
> >> >>> state = CONNECTED).  So right now I don't have a workaround and have
> >> >>> to kill
> >> >>> all the nodes and start over.
> >> >>>
> >> >>> On Thu, May 18, 2017 at 11:20 AM, Mark Payne <marka...@hotmail.com>
> >> >>> wrote:
> >> >>>>
> >> >>>> Hello,
> >> >>>>
> >> >>>> Just looking through this thread now. I believe that I understand
> the
> >> >>>> problem. I have updated the JIRA with details about what I think is
> >> >>>> the
> >> >>>> problem and a potential remedy for the problem.
> >> >>>>
> >> >>>> Thanks
> >> >>>> -Mark
> >> >>>>
> >> >>>> > On May 18, 2017, at 9:49 AM, Matt Gilman <
> matt.c.gil...@gmail.com>
> >> >>>> > wrote:
> >> >>>> >
> >> >>>> > Thanks for the additional details. They will be helpful when
> >> >>>> > working
> >> >>>> > the JIRA. All nodes, including the coordinator, heartbeat to the
> >> >>>> > active
> >> >>>> > coordinator. This means that the coordinator effectively
> heartbeats
> >> >>>> > to
> >> >>>> > itself. It appears, based on your log messages, that this is not
> >> >>>> > happening.
> >> >>>> > Because no heartbeats were receive from any node, the lack of
> >> >>>> > heartbeats
> >> >>>> > from the terminated node is not considered.
> >> >>>> >
> >> >>>> > Matt
> >> >>>> >
> >> >>>> > Sent from my iPhone
> >> >>>> >
> >> >>>> >> On May 18, 2017, at 8:30 AM, ddewaele <ddewa...@gmail.com>
> wrote:
> >> >>>> >>
> >> >>>> >> Found something interesting in the centos-b debug logging....
> >> >>>> >>
> >> >>>> >> after centos-a (the coordinator) is killed centos-b takes over.
> >> >>>> >> Notice how
> >> >>>> >> it "Will not disconnect any nodes due to lack of heartbeat" and
> >> >>>> >> how
> >> >>>> >> it still
> >> >>>> >> sees centos-a as connected despite the fact that there are no
> >> >>>> >> heartbeats
> >> >>>> >> anymore.
> >> >>>> >>
> >> >>>> >> 2017-05-18 12:41:38,010 INFO [Leader Election Notification
> >> >>>> >> Thread-2]
> >> >>>> >> o.apache.nifi.controller.FlowController This node elected
> Active
> >> >>>> >> Cluster
> >> >>>> >> Coordinator
> >> >>>> >> 2017-05-18 12:41:38,010 DEBUG [Leader Election Notification
> >> >>>> >> Thread-2]
> >> >>>> >> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor Purging old
> heartbeats
> >> >>>> >> 2017-05-18 12:41:38,014 INFO [Leader Election Notification
> >> >>>> >> Thread-1]
> >> >>>> >> o.apache.nifi.controller.FlowController This node has been
> elected
> >> >>>> >> Primary
> >> >>>> >> Node
> >> >>>> >> 2017-05-18 12:41:38,353 DEBUG [Heartbeat Monitor Thread-1]
> >> >>>> >> o.a.n.c.c.h.AbstractHeartbeatMonitor Received no new
> heartbeats.
> >> >>>> >> Will
> >> >>>> >> not
> >> >>>> >> disconnect any nodes due to lack of heartbeat
> >> >>>> >> 2017-05-18 12:41:41,336 DEBUG [Process Cluster Protocol
> Request-3]
> >> >>>> >> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor Received new
> heartbeat
> >> >>>> >> from
> >> >>>> >> centos-b:8080
> >> >>>> >> 2017-05-18 12:41:41,337 DEBUG [Process Cluster Protocol
> Request-3]
> >> >>>> >> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor
> >> >>>> >>
> >> >>>> >> Calculated diff between current cluster status and node cluster
> >> >>>> >> status as
> >> >>>> >> follows:
> >> >>>> >> Node: [NodeConnectionStatus[nodeId=centos-b:8080,
> state=CONNECTED,
> >> >>>> >> updateId=45], NodeConnectionStatus[nodeId=centos-a:8080,
> >> >>>> >> state=CONNECTED,
> >> >>>> >> updateId=42]]
> >> >>>> >> Self: [NodeConnectionStatus[nodeId=centos-b:8080,
> state=CONNECTED,
> >> >>>> >> updateId=45], NodeConnectionStatus[nodeId=centos-a:8080,
> >> >>>> >> state=CONNECTED,
> >> >>>> >> updateId=42]]
> >> >>>> >> Difference: []
> >> >>>> >>
> >> >>>> >>
> >> >>>> >> 2017-05-18 12:41:41,337 INFO [Process Cluster Protocol
> Request-3]
> >> >>>> >> o.a.n.c.p.impl.SocketProtocolListener Finished processing
> request
> >> >>>> >> 410e7db5-8bb0-4f97-8ee8-fc8647c54959 (type=HEARTBEAT,
> length=2341
> >> >>>> >> bytes)
> >> >>>> >> from centos-b:8080 in 3 millis
> >> >>>> >> 2017-05-18 12:41:41,339 INFO [Clustering Tasks Thread-2]
> >> >>>> >> o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at
> >> >>>> >> 2017-05-18
> >> >>>> >> 12:41:41,330 and sent to centos-b:10001 at 2017-05-18
> >> >>>> >> 12:41:41,339;
> >> >>>> >> send
> >> >>>> >> took 8 millis
> >> >>>> >> 2017-05-18 12:41:43,354 INFO [Heartbeat Monitor Thread-1]
> >> >>>> >> o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 1
> >> >>>> >> heartbeats
> >> >>>> >> in
> >> >>>> >> 93276 nanos
> >> >>>> >> 2017-05-18 12:41:46,346 DEBUG [Process Cluster Protocol
> Request-4]
> >> >>>> >> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor Received new
> heartbeat
> >> >>>> >> from
> >> >>>> >> centos-b:8080
> >> >>>> >> 2017-05-18 12:41:46,346 DEBUG [Process Cluster Protocol
> Request-4]
> >> >>>> >> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor
> >> >>>> >>
> >> >>>> >> Calculated diff between current cluster status and node cluster
> >> >>>> >> status as
> >> >>>> >> follows:
> >> >>>> >> Node: [NodeConnectionStatus[nodeId=centos-b:8080,
> state=CONNECTED,
> >> >>>> >> updateId=45], NodeConnectionStatus[nodeId=centos-a:8080,
> >> >>>> >> state=CONNECTED,
> >> >>>> >> updateId=42]]
> >> >>>> >> Self: [NodeConnectionStatus[nodeId=centos-b:8080,
> state=CONNECTED,
> >> >>>> >> updateId=45], NodeConnectionStatus[nodeId=centos-a:8080,
> >> >>>> >> state=CONNECTED,
> >> >>>> >> updateId=42]]
> >> >>>> >> Difference: []
> >> >>>> >>
> >> >>>> >>
> >> >>>> >>
> >> >>>> >>
> >> >>>> >> --
> >> >>>> >> View this message in context:
> >> >>>> >>
> >> >>>> >> http://apache-nifi-users-list.2361937.n4.nabble.com/Nifi-
> Cluster-fails-to-disconnect-node-when-node-was-killed-tp1942p1950.html
> >> >>>> >> Sent from the Apache NiFi Users List mailing list archive at
> >> >>>> >> Nabble.com.
> >> >>>>
> >> >>>
> >> >>
> >> >
> >
> >
>

Reply via email to