lordgamez opened a new pull request #1227: URL: https://github.com/apache/nifi-minifi-cpp/pull/1227
Originally the Jira ticket was about reowrking the engine queue to only allow the `maximum concurrent tasks` number of script engines to run at the same time. The problem with that was that even if the script engines were executed concurrently they use the same python interpreter with the global interpreter lock in place so no concurrency can take place. The only possibility for concurrency to use sub-interpreters which would be also problematic. First of all pybind11 does not support an interface for sub-interpreters, it is only available through the CPython API, but the documentation also warns about the caveats of the sub-interpreters here: https://pybind11.readthedocs.io/en/stable/advanced/embedding.html Secondly using the CPython API for sub-interpreters would require us to always know which thread we are running on and create separate sub-interpreters and thread states for each thread (in this case it would possibly not even allow us to run the python scripts from the main thread anymore). Managing the thread states and the sub-interpreters would be cumbersome in our architecture. Because all of the above problems I removed the python script queue from the ExecutePythonProcessor which will run the python scripts on a single thread. -------------------------------------------------- Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically main)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
