[
https://issues.apache.org/jira/browse/BEAM-3741?focusedWorklogId=164084&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-164084
]
ASF GitHub Bot logged work on BEAM-3741:
----------------------------------------
Author: ASF GitHub Bot
Created on: 08/Nov/18 20:59
Start Date: 08/Nov/18 20:59
Worklog Time Spent: 10m
Work Description: swegner commented on a change in pull request #6963:
[BEAM-3741] Proto changes for reporting backlog/splitting/finalizing bundles.
URL: https://github.com/apache/beam/pull/6963#discussion_r232058264
##########
File path: model/fn-execution/src/main/proto/beam_fn_api.proto
##########
@@ -573,36 +598,62 @@ message ProcessBundleProgressResponse {
// DEPRECATED (Required)
Metrics metrics = 1;
- // (Optional) Specifies that the bundle has been split since the last
- // ProcessBundleProgressResponse was sent.
- BundleSplit split = 2;
-
// (Required) The list of metrics or other MonitoredState
// collected while processing this bundle.
repeated MonitoringInfo monitoring_infos = 3;
+
+ // The list of currently active primary roots that are being
+ // executed. Required to be populated for PTransforms which can be split.
+ repeated BundleApplication primary_roots = 4;
}
+// Represents a request to the SDK to split a currently active bundle.
message ProcessBundleSplitRequest {
// (Required) A reference to an active process bundle request with the given
// instruction id.
string instruction_reference = 1;
- // Specifies that the runner would like the bundle to split itself using
- // BundleSplit, and give up some of the work that the bundle hasn't started
- // doing yet, so that it can be done in a separate bundle (perhaps in
- // parallel with the current bundle).
+ // (Required) Specifies that the Runner would like the bundle to split itself
+ // such that it performs no more work then the backlog specified for each
+ // PTransform. The interpretation of how much work should be processed is up
+ // to the PTransform.
//
- // The value is the fraction of unstarted work to keep. E.g. 0 means give up
- // as much as possible of unstarted work (e.g. checkpoint), 0.5 means give
- // up about half of the unstarted work, etc.
- // This is a hint and the value is approximate.
+ // For example, A backlog of "" tells the SDK to perform as little work as
+ // possible, effectively checkpointing when able. The remaining backlog
+ // will be relative to the backlog reported during processing.
//
- // The value is relative to the current scope of work of the bundle.
- google.protobuf.DoubleValue fraction_of_remainder = 2;
+ // If the backlog is unspecified for a PTransform, it would like that
Review comment:
can you update `it` to be more precise, i.e. "the runner requests that the
SDK process all data received for the PTransform"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 164084)
Time Spent: 6h 20m (was: 6h 10m)
> Proto changes for splitting over Fn API
> ---------------------------------------
>
> Key: BEAM-3741
> URL: https://issues.apache.org/jira/browse/BEAM-3741
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
> Reporter: Eugene Kirpichov
> Assignee: Eugene Kirpichov
> Priority: Major
> Fix For: 2.5.0
>
> Time Spent: 6h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
