I have been looking at how/whether NiFi can be used for mission-critical workflow orchestration. While there are various aspects to what this means, I am interested in how to modify/adapt NiFi to factor in task deadlines and task duration to do predictive scheduling. This is pretty different than the priority-queue-based scheduling, although it can coexist with it. When compute resources are limited this type of scheduling problem requires optimized execution plans. The plan must factor in task durations and deadlines. The plans must also factor in schedules of resource availability. In my use cases it is not about stream processing with lots of short-duration work items, it is more about one-shot orchestration of work-items that take minutes to hours.
I understand this is beyond the current scheduling model. However, if there was a way to “modularize” and create scheduler drop-ins that would open the door to experimentation without impacting the scheduling that is now available. Such a modularization would require: - A way to bundle up schedulers into a .nar (or something like it) - An internal scheduler factory that would identify the available schedulers upon NiFi startup. - A processor UI that would allow the user to select from among all the schedulers. I imagine this would look similar to what exists today, but be generalized. - A runtime UI-factory that could “reach into” the scheduler bundle to get a UI panel and display it inside the current processor UI. There might be two "flavors" to this UI. One is a detailed view that allows the user to enter and change parameters that the bundled scheduler understands. The other is a simpler, read-only, summary view of a subset of parameters (like desired completion time). - A way to serialize and deserialize the parameters displayed in that UI panel - A way to attach scheduler-dependent parameters to one of the structures passed to org.apache.nifi.controller.StandardProcessorNode so they could be inspected and used at runtime by the chosen scheduler. - And of course, modifications to StandardProcessorNode to identify the user-selected scheduler and interact with the scheduler bundle appropriately. Any ideas as to the most direct ways to achieve the above? Cheers, Naz Irizarry
