Hi,
I found that in `SdkHarness` do not  stop the `SdkWorker` when finish.  We
should add the logic for stop the `SdkWorker` in `SdkHarness`.  More detail
can be found [1].

There are two approaches to solve this issue:

Approach 1:  We can add a Fn API for teardown purpose and the runner will
teardown a specific bundle descriptor via this teardown Fn API during
disposing.
Approach 2: The control service termination could be seen as a signal and
once SDK harness receives this signal, the teardown of the bundle
descriptor will be performed.

More detail can be found in [2].

As the Approach 2, SDK harness could be shared between multiple executable
stages. The control service termination only occurs when all the executable
stages sharing the same SDK harness finished. This means that the teardown
of DoFns may not be executed immediately after an executable stage is
finished.

So, I prefer Approach 1. Welcome any feedback :)

Best,
Jincheng

[1]
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/sdk_worker.py
[2]
https://docs.google.com/document/d/1sCgy9VQPf9zVXKRquK8P6N4x7aB62GEO8ozkujRSHZg/edit?usp=sharing

Reply via email to