Hi Matt, I think this is great. I like the MVP set of features.
Considering use cases and potential maintenance needs, are there thoughts on including RESTART in the MVP for cases where agent UPDATES may be blocked for whatever reason ( I/O, agent problems, etc )? Does the risk of the RESTART verb being somewhat nebulous outweigh any benefits? Again, great stuff! Thanks, Marc On Thu, Nov 18, 2021 at 2:20 PM Matt Burgess <mattyb...@apache.org> wrote: > Hello all, > > I'd like to start a discussion about the current and future Command > and Control (C2) capabilities in terms of the NiFi/MiNiFi Java > codebase. The current implementation[1] is IMO not a pure reference > implementation of the C2 spec [2]. It has fairly limited capability > and uses the PullHttpChangeIngestor for the agent to ask for flow > updates, which makes any bidirectional communication difficult. Rather > than try to build out from what we currently have, I propose a new > implementation (deprecating and eventually removing the old one) that > is more generic, extensible, etc. This would include the following > options: > > - Renaming of the minifi-c2 modules and perhaps the release artifacts, > updating the documentation accordingly > - Creation of a top-level "c2" or "nifi-c2" module that includes an > API module, assembly, NAR, implementation, services bundle, etc. as > needed > - Adding/deleting/changing/using properties related to C2 configuration > - Changing the common runtime/loading code to look for a C2 client > implementation and instantiate that at startup, configuring it with > the appropriate properties and injecting the objects needed for C2 > capabilities > - Implementing a new C2 server that supports more of the C2 spec, and > making adding more of the commands easier going forward > > I propose the Minimum Viable Product (MVP) C2 Client would have the > following capabilities: > - A "Manifest Reporter" that can either push a manifest (a document > containing all the agent/instance capabilities such as available > processors and their metadata, e.g.) and/or respond to a DESCRIBE > MANIFEST request (see [2]) > - A "Heartbeat Reporter" to send HEARTBEAT messages and also to reply > with ACKNOWLEDGEMENT messages and dispatching any commands such as > UPDATE > - The Heartbeat Reporter is analogous to the PullHttpChangeIngestor in > the old system, but is much more capable > - Supports HTTPS > > I propose the MVP C2 Server would have the following capabilities: > - Handles manifest messages (push and/or pull) aka DESCRIBE MANIFEST > - Handles the HEARTBEAT / UPDATE / ACK exchange, perhaps initially > using the same approach as the old C2 system (loading the flow from a > file for the UPDATE command) but would be extensible/configurable > - Supports HTTPS > > This should give us not only feature parity with the old C2 system, > but also presents a framework to build out the rest of the C2 design > and add other capabilities. I should note that this proposal is not > specific to MiNiFi Java. It should aim to align and use the same > protocol(s) as MiNiFi C++ [3], and should allow us to perhaps add C2 > capabilities to NiFi itself in the future. We could track the proposal > and progress at [4] as I believe that's what the page is intended for. > > I'd like to hear your thoughts on these proposals, and I'm happy to > answer any questions or clarify as necessary. Also as this proceeds > I'm happy to write up any Jira epics/cases/etc. and do the initial > project structure and API setup work so others can dig in. > > Regards, > Matt > > [1] https://github.com/apache/nifi/blob/main/minifi/minifi-c2/README.md > [2] https://cwiki.apache.org/confluence/display/MINIFI/C2+Design > [3] https://github.com/apache/nifi-minifi-cpp/blob/main/C2.md > [4] > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177048430&flashId=624428108 >
