Approach 2 is currently the suggested approach[1] for DoFn's to shutdown. Note that SDK harnesses can terminate instances any time they want and start new instances anytime as well.
Why do you want to expose this logic so that Runners could control it? 1: https://docs.google.com/document/d/1n6s3BOxOPct3uF4UgbbI9O9rpdiKWFH9R6mtVmR7xp0/edit# On Mon, Oct 21, 2019 at 4:27 AM jincheng sun <sunjincheng...@gmail.com> wrote: > Hi, > I found that in `SdkHarness` do not stop the `SdkWorker` when finish. We > should add the logic for stop the `SdkWorker` in `SdkHarness`. More detail > can be found [1]. > > There are two approaches to solve this issue: > > Approach 1: We can add a Fn API for teardown purpose and the runner will > teardown a specific bundle descriptor via this teardown Fn API during > disposing. > Approach 2: The control service termination could be seen as a signal and > once SDK harness receives this signal, the teardown of the bundle > descriptor will be performed. > > More detail can be found in [2]. > > As the Approach 2, SDK harness could be shared between multiple executable > stages. The control service termination only occurs when all the executable > stages sharing the same SDK harness finished. This means that the teardown > of DoFns may not be executed immediately after an executable stage is > finished. > > So, I prefer Approach 1. Welcome any feedback :) > > Best, > Jincheng > > [1] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/sdk_worker.py > [2] > https://docs.google.com/document/d/1sCgy9VQPf9zVXKRquK8P6N4x7aB62GEO8ozkujRSHZg/edit?usp=sharing >
