[
https://issues.apache.org/jira/browse/BEAM-3741?focusedWorklogId=164081&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-164081
]
ASF GitHub Bot logged work on BEAM-3741:
----------------------------------------
Author: ASF GitHub Bot
Created on: 08/Nov/18 20:59
Start Date: 08/Nov/18 20:59
Worklog Time Spent: 10m
Work Description: swegner commented on a change in pull request #6963:
[BEAM-3741] Proto changes for reporting backlog/splitting/finalizing bundles.
URL: https://github.com/apache/beam/pull/6963#discussion_r232053694
##########
File path: model/fn-execution/src/main/proto/beam_fn_api.proto
##########
@@ -184,55 +187,72 @@ message ProcessBundleDescriptor {
org.apache.beam.model.pipeline.v1.ApiServiceDescriptor
state_api_service_descriptor = 7;
}
-// Represents a partition of the bundle into two bundles: a "primary" and
-// a "residual", with the following properties:
-// - The work in primary and residual doesn't overlap, and combined, adds up
-// to the work in the current bundle if the split hadn't happened.
-// - The current bundle, if it keeps executing, will have done none of the
-// work under residual roots.
-// - The current bundle, if no further splits happen, will have done exactly
-// the work under primary_roots.
-// For more rigorous definitions see https://s.apache.org/beam-breaking-fusion
-message BundleSplit {
- // One of the root applications specifying the scope of work for a bundle.
- message Application {
- // (Required) The primitive transform to which to pass the element
- string ptransform_id = 1;
+// One of the applications specifying the scope of work for a bundle.
+message BundleApplication {
+ // (Required) The primitive transform to which to pass the element
+ string ptransform_id = 1;
+
+ // (Required) Name of the transform's input to which to pass the element.
+ string input_id = 2;
- // (Required) Name of the transform's input to which to pass the element.
- string input_id = 2;
+ // (Required) The encoded element to pass to the transform.
+ bytes element = 3;
- // (Required) The encoded element to pass to the transform.
- bytes element = 3;
+ // Lower bound on timestamps of elements that this PTransform
+ // will produce into each of its output PCollections when invoked on this
+ // element and input. The map is keyed by the local output name of the
+ // PTransform.
+ map<string, google.protobuf.Timestamp> output_watermarks = 4;
- // Approximate lower bounds on timestamps of elements that this PTransform
- // will produce into each of its output PCollections, when invoked on this
- // element. Keyed by the transform's local output name.
- map<string, int64> output_watermarks = 4;
+ // Represents an estimate for the amount of currently outstanding work.
+ message Backlog {
+ // This informs Runners on how to aggregate the backlog
+ // being reported across multiple active bundles. Backlogs
+ // are aggregated using the set of partitions.
+ //
+ // For example SplittableDoFn's which consume elements from:
+ // * a globally shared resource such as a Pubsub queue should set this
+ // to “”.
+ // * a shared partitioned resource should use the partition identifier.
+ // * a uniquely partitioned resource such as a file range should set this
to
+ // file name + start offset.
+ bytes partition = 1;
Review comment:
Should this be a `String` (versus `bytes`?) The documentation refers to the
empty string `""`, and it seems consistent with other identifiers which are
strings.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 164081)
Time Spent: 5h 50m (was: 5h 40m)
> Proto changes for splitting over Fn API
> ---------------------------------------
>
> Key: BEAM-3741
> URL: https://issues.apache.org/jira/browse/BEAM-3741
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
> Reporter: Eugene Kirpichov
> Assignee: Eugene Kirpichov
> Priority: Major
> Fix For: 2.5.0
>
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
