[
https://issues.apache.org/jira/browse/BEAM-8335?focusedWorklogId=386157&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-386157
]
ASF GitHub Bot logged work on BEAM-8335:
----------------------------------------
Author: ASF GitHub Bot
Created on: 12/Feb/20 19:21
Start Date: 12/Feb/20 19:21
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10826: [BEAM-8335]
Modify the TestStreamPayload to accept an argument of output_tags and…
URL: https://github.com/apache/beam/pull/10826#discussion_r378459440
##########
File path: model/interactive/src/main/proto/beam_interactive_api.proto
##########
@@ -40,18 +40,22 @@ import "google/protobuf/timestamp.proto";
message TestStreamFileHeader {
// The PCollection tag this stream is associated with.
string tag = 1;
+
+ // The file format version. This is used to ensure backwards compatibility
+ // when decoding from file.
+ int32 version = 2;
}
// A record is a recorded element that a source produced. Its function is to
// give enough information to create a faithful recreation of the original
// stream of data.
message TestStreamFileRecord {
oneof recorded_event {
- // The recorded element with its event timestamp (when it was produced).
- org.apache.beam.model.pipeline.v1.TestStreamPayload.TimestampedElement
element = 1;
+ // The recorded bundle with its event timestamp (when it was produced).
+ org.apache.beam.model.pipeline.v1.TestStreamPayload.Event.AddElements
element_event = 1;
// Indicating the output watermark of the source changed.
- google.protobuf.Timestamp watermark = 2;
+ org.apache.beam.model.pipeline.v1.TestStreamPayload.Event.AdvanceWatermark
watermark_event = 2;
}
// The wall-time timestamp of either the new element or watermark change.
Review comment:
You would still be able to do that "compression" yourself and only add a
single AdvanceProcessingTime yet still have the flexibility to record multiple
AdvanceProcessingTime events if they were ever necessary.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 386157)
> Add streaming support to Interactive Beam
> -----------------------------------------
>
> Key: BEAM-8335
> URL: https://issues.apache.org/jira/browse/BEAM-8335
> Project: Beam
> Issue Type: Improvement
> Components: runner-py-interactive
> Reporter: Sam Rohde
> Assignee: Sam Rohde
> Priority: Major
> Time Spent: 56h 40m
> Remaining Estimate: 0h
>
> This issue tracks the work items to introduce streaming support to the
> Interactive Beam experience. This will allow users to:
> * Write and run a streaming job in IPython
> * Automatically cache records from unbounded sources
> * Add a replay experience that replays all cached records to simulate the
> original pipeline execution
> * Add controls to play/pause/stop/step individual elements from the cached
> records
> * Add ability to inspect/visualize unbounded PCollections
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
