[
https://issues.apache.org/jira/browse/BEAM-11104?focusedWorklogId=765683&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-765683
]
ASF GitHub Bot logged work on BEAM-11104:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 03/May/22 19:55
Start Date: 03/May/22 19:55
Worklog Time Spent: 10m
Work Description: jrmccluskey commented on code in PR #17533:
URL: https://github.com/apache/beam/pull/17533#discussion_r864151749
##########
CHANGES.md:
##########
@@ -66,6 +66,7 @@
## New Features / Improvements
* 'Manage Clusters' JupyterLab extension added for users to configure usage of
Dataproc clusters managed by Interactive Beam (Python)
([BEAM-14130](https://issues.apache.org/jira/browse/BEAM-14130)).
+* Go SDK users may now write self-checkpointing Splittable DoFns to read from
streaming sources. **Note: this feature is not yet fully validated and should
be treated as experimental in this release.**
([BEAM-11104](https://issues.apache.org/jira/browse/BEAM-11104))
Review Comment:
I updated the ticket to have that information. Largely just the lack of an
integration test for it right now, will be poking around at that this week
since Impulses apparently count as DataSources. We should be able to mock
together a somewhat generic test case assuming the runners play nice and
terminate the pipeline when appropriate.
Issue Time Tracking
-------------------
Worklog Id: (was: 765683)
Time Spent: 16h 40m (was: 16.5h)
> [Go SDK] DoFn Self Checkpointing
> --------------------------------
>
> Key: BEAM-11104
> URL: https://issues.apache.org/jira/browse/BEAM-11104
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-go
> Reporter: Robert Burke
> Assignee: Jack McCluskey
> Priority: P3
> Time Spent: 16h 40m
> Remaining Estimate: 0h
>
> Allow SplittableDoFns to self checkpoint.
> Design doc:
> [https://docs.google.com/document/d/1_JbzjY9JR07ZK5v7PcZevUfzHPsqwzfV7W6AouNpMPk/edit?usp=sharing]
>
> Feature is written E2E and users will be able to return ProcessContinuations
> from SDFs as of 2.39.0 but the full behavior has not been fully validated. An
> integration test that validates self-checkpointing is working as-intended
> will need to be written and passing before the feature is no longer
> considered experimental and this ticket is marked as resolved.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
