[ https://issues.apache.org/jira/browse/BEAM-6777?focusedWorklogId=258283&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-258283 ]
ASF GitHub Bot logged work on BEAM-6777: ---------------------------------------- Author: ASF GitHub Bot Created on: 12/Jun/19 02:17 Start Date: 12/Jun/19 02:17 Worklog Time Spent: 10m Work Description: robinyqiu commented on pull request #8827: [BEAM-6777] Let HealthzServlet respond actual health information of SDK harnesses URL: https://github.com/apache/beam/pull/8827#discussion_r292724536 ########## File path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SdkHarnessRegistries.java ########## @@ -107,6 +108,17 @@ public void unregisterWorkerClient(FnApiControlClient controlClient) { workers.remove(worker); } LOG.info("Unregistered Control client {}", worker != null ? worker.getWorkerId() : null); + + // unregisterWorkerClient() will be called only when the connection between SDK harness and + // runner harness is broken or SDK harness respond to runner harness with an error. In either + // case, the SDK should be marked as unhealthy. + sdkHarnessesAreHealthy.set(false); Review comment: Taking progress update into consideration as an indicator of SDK harness health is definitely something worth exploring, but before that I want to focus the current discussion (see the bug) on whether we should treat bundle execution failure as SDK harness unhealthy. If not, what do we need to do if the bundle keeps failing? Do we simply retry indefinitely locally? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking ------------------- Worklog Id: (was: 258283) Time Spent: 6.5h (was: 6h 20m) > SDK Harness Resilience > ---------------------- > > Key: BEAM-6777 > URL: https://issues.apache.org/jira/browse/BEAM-6777 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow > Reporter: Sam Rohde > Assignee: Yueyang Qiu > Priority: Major > Time Spent: 6.5h > Remaining Estimate: 0h > > If the Python SDK Harness crashes in any way (user code exception, OOM, etc) > the job will hang and waste resources. The fix is to add a daemon in the SDK > Harness and Runner Harness to communicate with Dataflow to restart the VM > when stuckness is detected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)