[ https://issues.apache.org/jira/browse/BEAM-8618?focusedWorklogId=378253&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-378253 ]
ASF GitHub Bot logged work on BEAM-8618: ---------------------------------------- Author: ASF GitHub Bot Created on: 28/Jan/20 14:06 Start Date: 28/Jan/20 14:06 Worklog Time Spent: 10m Work Description: mxm commented on pull request #10655: [BEAM-8618] Tear down unused DoFns periodically in Python SDK harness. URL: https://github.com/apache/beam/pull/10655#discussion_r371817547 ########## File path: sdks/python/apache_beam/runners/worker/sdk_worker.py ########## @@ -280,6 +283,7 @@ def get(self, instruction_id, bundle_descriptor_id): try: # pop() is threadsafe processor = self.cached_bundle_processors[bundle_descriptor_id].pop() + self.last_access_time[bundle_descriptor_id] = time.time() except IndexError: Review comment: This won't update the access time when we first create the processor in the except block. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking ------------------- Worklog Id: (was: 378253) Time Spent: 40m (was: 0.5h) > Tear down unused DoFns periodically in Python SDK harness > --------------------------------------------------------- > > Key: BEAM-8618 > URL: https://issues.apache.org/jira/browse/BEAM-8618 > Project: Beam > Issue Type: Improvement > Components: sdk-py-harness > Reporter: sunjincheng > Assignee: sunjincheng > Priority: Major > Fix For: 2.20.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Per the discussion in the ML, detail can be found [1], the teardown of DoFns > should be supported in the portability framework. It happens at two places: > 1) Upon the control service termination > 2) Tear down the unused DoFns periodically > The aim of this JIRA is to add support for tear down the unused DoFns > periodically in Python SDK harness. > [1] > https://lists.apache.org/thread.html/0c4a4cf83cf2e35c3dfeb9d906e26cd82d3820968ba6f862f91739e4@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)