[
https://issues.apache.org/jira/browse/NIP-26?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18075274#comment-18075274
]
Kevin Doran commented on NIP-26:
--------------------------------
Thanks [~markap14] - that seems reasonable to me.
> Automatic Scheduling Strategy for Processors
> --------------------------------------------
>
> Key: NIP-26
> URL: https://issues.apache.org/jira/browse/NIP-26
> Project: NiFi Improvement Proposal
> Issue Type: New Feature
> Reporter: Mark Payne
> Assignee: Mark Payne
> Priority: Major
>
> h3. Motivation
> NiFi currently requires users to manually configure the number of concurrent
> tasks for each processor. Choosing the right value is difficult -- too few
> tasks leaves throughput on the table, while too many can cause excessive
> contention or overwhelm downstream processors. Additionally, what is ideal in
> one environment may not be ideal in another environment with different
> resources.
> We should introduce an "Automatic" scheduling strategy that dynamically
> adjusts the number of concurrent tasks for a processor based on runtime
> conditions such as queue depth, backpressure, throughput, and CPU load. This
> removes the need for manual tuning and allows the system to adapt as
> workloads change over time.
> h3. NiFi API Changes
> * Introduce a new {{SchedulingStrategy.AUTO}} enum value. When selected, the
> framework manages concurrency automatically; the user does not configure a
> concurrent task count, run schedule, or run duration.
> * Introduce a new {{@AllowsAutoScheduling}} annotation with a {{boolean
> value()}} defaulting to {{{}true{}}}. Processors that allocate a fixed number
> of resources per concurrent task at schedule time (e.g., one Kafka consumer
> per task, one script engine per task) should use
> {{@AllowsAutoScheduling(false)}} to opt out. Processors annotated this way
> will not show the Automatic option in the UI.
> h3. REST API Changes
> * {{ProcessorDTO}} gains a {{supportsAutoScheduling}} boolean field,
> populated from the annotation. The REST API validates that {{AUTO}} cannot be
> set on processors that do not support it.
> * {{ProcessorConfigDTO}} default concurrent task and scheduling period maps
> conditionally include or exclude the {{AUTO}} entry based on the processor's
> annotation.
> h3. Framework Changes
> * A new {{VirtualThreadSchedulingAgent}} replaces the thread-pool-based
> scheduling agent. All processors (regardless of strategy) run on virtual
> threads, using a semaphore to bound the number of tasks that can be run
> concurrently (effectively, applying a cap on Virtual Threads in much the same
> way that the thread pool currently does). The virtual threads are beneficial
> here because it allows for much cleaner addition and removal of tasks/threads
> for a given processor. More generally, though, this has been desirable for a
> while because it simplifies a lot of the scheduling logic and pins Processors
> to specific threads, which makes thread dumps clearer and enables the use of
> ThreadLocal that historically we've avoided in Processors.
> * For AUTO processors, the Processor starts with a single virtual thread and
> periodically evaluates whether to scale up or down using a
> {{VirtualThreadScaler}} interface. This provides a clear separation between
> the scaling logic and the scheduling logic. The initial implementation would
> use a queue-based heuristic (examining inbound queue depth, outbound
> backpressure ratio, and throughput to ensure that increasing threads actually
> helps). Additionally, it should consider CPU load so that when adding 1
> thread helps but we're already over-budget on CPU we don't keep adding. We'd
> also introduce a cap on the number of threads that any given processor can be
> given - probably configurable via nifi.properties with a max of 12 threads.
> * {{ProcessorContext.getMaxConcurrentTasks()}} should return the max number
> of threads available for the Processor in AUTO mode (12 by default or 1 for
> @TriggerSerially processors)
> * Diagnostic dump that handles the thread dump will need a slight
> modifications because the existing MXBean does not include virtual threads;
> we'd need to change to the {{{}HotSpotDiagnosticMXBean{}}}.
> h3. UI Changes
> * The Scheduling Strategy dropdown gains an "Automatic (Experimental)"
> option. When selected, the Concurrent Tasks, Run Schedule, and Run Duration
> fields are hidden since they are managed by the framework.
> * The option is filtered out for processors annotated with
> {{{}@AllowsAutoScheduling(false){}}}.
> h3. nifi.properties Configuration
> * {{nifi.processor.max.concurrent.tasks}} -- upper bound on concurrent tasks
> per processor for both manual and automatic scheduling. We could potentially
> have a property that applies only to those Processors in Auto mode. However,
> we frequently see users configuring values that are drastically too high, in
> the hundreds or more, and this leads to performance degradation and other
> problems. So it makes sense to just apply this to all processors regardless
> of their scheduling strategy instead of going out of our way to apply it only
> to Auto mode.
>
> Note that this would be introduced as an "Experimental" feature and would
> clearly be labeled as such in the UI.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
