tvalentyn commented on code in PR #35234:
URL: https://github.com/apache/beam/pull/35234#discussion_r2139191584
##########
sdks/python/apache_beam/ml/inference/vllm_inference.py:
##########
@@ -114,28 +114,30 @@ def __init__(self, model_name: str, vllm_server_kwargs:
dict[str, str]):
self._server_started = False
self._server_process = None
self._server_port: int = -1
+ self._server_process_lock = threading.RLock()
Review Comment:
When `large_model=true`, multiple processes share the server through a
single server handle shared across processes, and only one process is
responsible for managing the handle. The cross-process lock should not be
necessary.
The problem is that right now we call `check_connectivity` from an async
method:
https://github.com/apache/beam/blob/de76cfa16d36e2cb8dd1a1092710f0cd477e0f8d/sdks/python/apache_beam/ml/inference/vllm_inference.py#L219
and `check_connectivity` happens to also start the server if the process
already terminated, so as a result we create many processes at once (see
internal: b/422577332), and overwrite `self._server_process`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@beam.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
