Hi Luis,
I've observed similar behavior working with the python processors.
<SNIP>
> 1. Everytime I modify the python code of a processor and stop it, all
> the dependencies are re-installed. Doesn't matter if I just changed code
> or added any new library. Is that ok ?
</SNIP>
The modification behavior seems like it stems from how the
PythonProcessorProxy handles reloads in the onSchedule [1] and checks
the lastModified of the module file [2]. If the reload is triggered it
kicks off the code in the Extension manager [3]. So modifying the files
triggers the reload. It does look like there might be opportunities to
improve this but given dependence can be managed in the source files it
seems like it would likely not be super simple.
<SNIP>
> 2. When I stop and/or disable the custom processor, the os process
> doesn't die, it stays running or in some state.
</SNIP>
The stopping seems to be tied to the onStopped [4] function that is
bound to a Python class that implements the onStopped behavior [5]. I
don't see anything in there about killing the process that appears to be
managed in the PythonProcess class [6] and doesn't look like it's called
until shutting down the FlowController [7] which seems to be more of an
application shutdown event. Are you seeing issues with leaving the
processors running? Seems like there may be a tradeoff here between
leaving the processes running and the cost of starting and stopping them
each time. Would be interesting to hear if this is causing a specific
issue in the code you're running.
Overall I don't think you're seeing anything I haven't also seen in my
experience with the python processors; the best documentation I've found
is [8]. Might be interesting to consider if the implementation is
static enough to document or if it's likely to continue to rapidly
evolve in which case document it might be premature.
Hope this helps!
- Bob Paulin
[1]
https://github.com/apache/nifi/blob/a56572d3b20222ebae632d35fd132686d536a3d8/nifi-extension-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/main/java/org/apache/nifi/python/processor/PythonProcessorProxy.java#L319
[2]
https://github.com/apache/nifi/blob/a56572d3b20222ebae632d35fd132686d536a3d8/nifi-extension-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/main/java/org/apache/nifi/py4j/StandardPythonProcessorBridge.java#L181
[3]
https://github.com/apache/nifi/blob/a56572d3b20222ebae632d35fd132686d536a3d8/nifi-extension-bundles/nifi-py4j-bundle/nifi-python-framework/src/main/python/framework/ExtensionManager.py#L108
[4]
https://github.com/apache/nifi/blob/a56572d3b20222ebae632d35fd132686d536a3d8/nifi-extension-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/main/java/org/apache/nifi/python/processor/PythonProcessorProxy.java#L333
[5]
https://github.com/apache/nifi/blob/a56572d3b20222ebae632d35fd132686d536a3d8/nifi-extension-bundles/nifi-py4j-bundle/nifi-python-framework/src/main/python/framework/PythonProcessorAdapter.py#L109
[6]
https://github.com/apache/nifi/blob/a56572d3b20222ebae632d35fd132686d536a3d8/nifi-extension-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/main/java/org/apache/nifi/py4j/PythonProcess.java#L389
[7]
https://github.com/apache/nifi/blob/a56572d3b20222ebae632d35fd132686d536a3d8/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FlowController.java#L1562
[8]
https://nifi.apache.org/documentation/nifi-2.0.0-M3/html/python-developer-guide.html
On 2024/06/08 03:45:23 Luis Carmona wrote:
> Hi guys,
>
> according to what I understand from the documentation, every time I
> create a python processor and manage to add it to nifi it should be
> started from a new instance of the selected python interpreter. To be
> more accurate, it should be started from it's own new virtual
> environment - created under .../work/..., and this python should start
> the python module as a new process.
>
> But there are a couple of things I'm doing wrong or I am not
> understanding well from the documentation, because:
>
> 1. Everytime I modify the python code of a processor and stop it, all
> the dependencies are re-installed. Doesn't matter if I just changed code
> or added any new library. Is that ok ?
>
> 2. When I stop and/or disable the custom processor, the os process
> doesn't die, it stays running or in some state. The sequence I think
> happens is watching Linux htop:
>
> - Java starts a controller (python)
>
> - The custom python component starts the python controller as well ¿?
>
> I am going nuts with some strange behavior with this cause it is not
> easy to understand if the process hung up, or I have a bug, or the
> processor hasn't actually been reloaded and restarted.
>
>
> Is there any document about the mechanics of this I can read ?
>
>
> Thanks in advance
>
>
> LC
>
>
>
>
