" This would align it more to the way NiFi works as well, allowing C2 to be used for flow control and metadata exchange rather than process control."
This particularly caught my eye and is something that makes sense from an evolutionary perspective. I've previously viewed command and control as a mechanism to control the agent processes and their flows. >From an operational perspective, I think that agent process control would be nice to have (if not a necessity for some ); however, if C2 is scoped to metadata and flow control at least -- through the MVP or permanently -- then the conversation is simpler and I think your proposal is great. Thanks for the additional insight, Matt. Thanks, Marc On Thu, Nov 18, 2021 at 4:37 PM Matt Burgess <mattyb...@apache.org> wrote: > Marc, > > That's a great point and reminds me that I missed two whole sections > of the proposal: 1) The refactor of minifi-bootstrap, and 2) the > standardization of serialized flow control to VersionedFlowSnapshot > (i.e. JSON not XML or YAML). > > 1) Currently it's the bootstrap process that reaches out to the C2 > server and handles the commands such as UPDATE and RESTART. In fact, > UPDATE updates the flow on-disk and RESTART then kills and restarts > the MiNiFi process. I think with the time that's passed since its > inception, it has kind of gotten bloated and has taken on capabilities > probably best left to a separate module, since those things make it > inconsistent with the way NiFi bootstrap works . In that vein I'd > propose (at some point, not necessarily MVP but that'd be great) that > we standardize the bootstraps as much as possible, and put all the C2 > stuff into the C2 modules (i.e. the C2 Client in the case of > minifi-bootstrap). I vaguely alluded to that when I said the C2 client > would need access to the internal objects, but that glossed WAY over > the actual implication, which is that the C2 client module will be the > thing talking to the C2 server to handle all the C2 messages including > UPDATE, and I think RESTART might be replaced with RELOAD or absorbed > into UPDATE, but happy to discuss that as well. > > If there are issues such as updating an agent/instance with > configuration properties, I anticipate a new proposal and design to > cover that, but I invite current C2 Server users to jump in here > please. As it stands I believe we haven't implemented enough of the C2 > capabilities to be affected, unless the C2 server's flow update file > includes configurable property updates. Otherwise I would think just a > flow update would suffice, and could be done without restarting the > process itself. This would align it more to the way NiFi works as > well, allowing C2 to be used for flow control and metadata exchange > rather than process control. > > 2) IIRC, the "config.yml' approach came from two use cases, to be able > to update the agent process/configuration itself and to make it easier > to "design" a flow by having more user-friendly names for the various > configuration and component properties without a UI designer. These > days, the minifi-toolkit allows the user to convert a flow from > flow.xml.gz to a config.yml, and I'm aware of at least one vendor > product for flow design & publish via a GUI. Between the > aforementioned lack of a need to update the agent process, and ongoing > work [1] to standardize the flow definition for NiFi, Registry, > Stateless, etc. to VersionedFlowSnapshot, I think MiNiFi should follow > suit. I can't speak to the C++ side of things but perhaps those more > familiar with that project can weigh in. > > My original email turned out to be more focused on the MVP products, > but these additional aspects (as well as any others the community > would like to offer) should be discussed too. If these are not > necessary for MVP, perhaps we can get the work going on the API and > MVP and continue the discussions on the Wiki and/or mailing list as > folks want to contribute the extra goodies :) > > Thanks, > Matt > > [1] https://issues.apache.org/jira/browse/NIFI-9069 > > On Thu, Nov 18, 2021 at 4:07 PM Marc Parisi <phroc...@apache.org> wrote: > > > > Hi Matt, > > > > I think this is great. I like the MVP set of features. > > > > Considering use cases and potential maintenance needs, are there thoughts > > on including RESTART in the MVP for cases where agent UPDATES may be > > blocked for whatever reason ( I/O, agent problems, etc )? > > > > Does the risk of the RESTART verb being somewhat nebulous outweigh any > > benefits? > > > > Again, great stuff! > > > > Thanks, > > Marc > > > > > > On Thu, Nov 18, 2021 at 2:20 PM Matt Burgess <mattyb...@apache.org> > wrote: > > > > > Hello all, > > > > > > I'd like to start a discussion about the current and future Command > > > and Control (C2) capabilities in terms of the NiFi/MiNiFi Java > > > codebase. The current implementation[1] is IMO not a pure reference > > > implementation of the C2 spec [2]. It has fairly limited capability > > > and uses the PullHttpChangeIngestor for the agent to ask for flow > > > updates, which makes any bidirectional communication difficult. Rather > > > than try to build out from what we currently have, I propose a new > > > implementation (deprecating and eventually removing the old one) that > > > is more generic, extensible, etc. This would include the following > > > options: > > > > > > - Renaming of the minifi-c2 modules and perhaps the release artifacts, > > > updating the documentation accordingly > > > - Creation of a top-level "c2" or "nifi-c2" module that includes an > > > API module, assembly, NAR, implementation, services bundle, etc. as > > > needed > > > - Adding/deleting/changing/using properties related to C2 configuration > > > - Changing the common runtime/loading code to look for a C2 client > > > implementation and instantiate that at startup, configuring it with > > > the appropriate properties and injecting the objects needed for C2 > > > capabilities > > > - Implementing a new C2 server that supports more of the C2 spec, and > > > making adding more of the commands easier going forward > > > > > > I propose the Minimum Viable Product (MVP) C2 Client would have the > > > following capabilities: > > > - A "Manifest Reporter" that can either push a manifest (a document > > > containing all the agent/instance capabilities such as available > > > processors and their metadata, e.g.) and/or respond to a DESCRIBE > > > MANIFEST request (see [2]) > > > - A "Heartbeat Reporter" to send HEARTBEAT messages and also to reply > > > with ACKNOWLEDGEMENT messages and dispatching any commands such as > > > UPDATE > > > - The Heartbeat Reporter is analogous to the PullHttpChangeIngestor in > > > the old system, but is much more capable > > > - Supports HTTPS > > > > > > I propose the MVP C2 Server would have the following capabilities: > > > - Handles manifest messages (push and/or pull) aka DESCRIBE MANIFEST > > > - Handles the HEARTBEAT / UPDATE / ACK exchange, perhaps initially > > > using the same approach as the old C2 system (loading the flow from a > > > file for the UPDATE command) but would be extensible/configurable > > > - Supports HTTPS > > > > > > This should give us not only feature parity with the old C2 system, > > > but also presents a framework to build out the rest of the C2 design > > > and add other capabilities. I should note that this proposal is not > > > specific to MiNiFi Java. It should aim to align and use the same > > > protocol(s) as MiNiFi C++ [3], and should allow us to perhaps add C2 > > > capabilities to NiFi itself in the future. We could track the proposal > > > and progress at [4] as I believe that's what the page is intended for. > > > > > > I'd like to hear your thoughts on these proposals, and I'm happy to > > > answer any questions or clarify as necessary. Also as this proceeds > > > I'm happy to write up any Jira epics/cases/etc. and do the initial > > > project structure and API setup work so others can dig in. > > > > > > Regards, > > > Matt > > > > > > [1] > https://github.com/apache/nifi/blob/main/minifi/minifi-c2/README.md > > > [2] https://cwiki.apache.org/confluence/display/MINIFI/C2+Design > > > [3] https://github.com/apache/nifi-minifi-cpp/blob/main/C2.md > > > [4] > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177048430&flashId=624428108 > > > >
