joewitt commented on PR #11164: URL: https://github.com/apache/nifi/pull/11164#issuecomment-5640200307
1. Please keep a system-test path on standard scheduling (or AUTO on Java 21). What I am asking: the shipped default is now AUTO, which means Java 21 still runs the existing platform-thread scheduler. But every system-test nifi.properties on this branch sets nifi.scheduling.strategy=VIRTUAL (default, pythonic, both cluster nodes). VirtualThreadStartStopCycleIT also forces VIRTUAL. So CI on Java 21 will exercise the new engine, while a real Java 21 install with an untouched config will exercise the old one. There is no system-test graph that starts, stops, clusters, or fails over on STANDARD or on AUTO as Java 21 will actually resolve it. Why it matters: we now have two production schedulers for a while. Virtual-thread coverage on Java 21 CI is useful bake — I would keep that. What we lose is regression coverage of the path most Java 21 operators will actually run, plus the resolution of AUTO itself (unit-tested only). Start/stop, run-once, terminate, clustered primary-node, and “disable the controller service then delete it” are exactly the races that differ between a delayed-future pool and a long-lived virtual thread. If those only run under VIRTUAL, we will not notice if STANDARD was accidentally broken by the wiring change, and we will not notice if AUTO on Java 21 fails to select standard scheduling in a real process. A small matrix is enough: leave most shards on VIRTUAL so the new engine gets the heat, and keep at least one Java 21 system-test profile on STANDARD or unset/AUTO. The AUTO resolver already has a unit test; an integration assertion that the running node logged “platform thread pool” on Java 21 and “virtual threads” on Java 25 would close the loop. 2. Please shrink the leftover FlowEngine when virtual threads are in charge of components. What I am asking: when VIRTUAL (or AUTO on Java 25) is selected, Processors and Reporting Tasks no longer run on the timer-driven FlowEngine. That pool is still created at nifi.flowcontroller.maxTimerDrivenThreadCount, and raising max threads in the UI still calls setCorePoolSize on it and resizes the virtual-thread semaphore. The “active timer-driven thread” metric is the sum of the two. Why it matters: today, max threads = 200 means about 200 platform threads that are the component workers. After this change, max threads = 200 means 200 semaphore permits for component work plus a 200-thread platform pool that is only used for a handful of framework periodic tasks (status history snapshot, remote process group refresh, registry sync, Python extension discovery, analytics predictions). Those jobs do not need 200 workers. Pierre raised this earlier, and the earlier branch had the leftover pool as a small framework engine. The current rewrite puts the old size back. That has three practical effects. Operators who raise max threads to feed a large canvas also stand up a large idle platform pool they no longer need. The thread-count statistic is no longer “how many component tasks are running”; it can show framework pool activity plus permit holders, so a quiet canvas can still look busy, or a busy canvas can look busier than the cap. Shutdown waits on the platform pool first, then the virtual-thread executor, so a large leftover pool can eat the graceful-stop budget before component threads are waited on. The max-thread property should mean one thing: how many Processor/Reporting Task invocations may run at once. Under virtual-thread scheduling that is the semaphore. The leftover FlowEngine should be a small, fixed framework pool (something like the 8 you had locally before), not resized with max threads, and the active-thread metric should not add the two budgets together as if they were the same cap. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
