junaiddshaukat opened a new pull request, #39273:
URL: https://github.com/apache/beam/pull/39273
Adds a translator for Beam's Flatten primitive (`beam:transform:flatten:v1`)
--
the union of N input PCollections into one. Part of #18479; the last
primitive
needed before the first PAssert-based `@ValidatesRunner` test (PAssert's
`GroupGlobally` uses `GBK + Flatten`, no side inputs).
What's here:
- `FlattenProcessor` -- forwards data through, and owns its output watermark
the
way GroupByKey does: a `WatermarkManager` over the input branches,
emitting its
own single-source `(0 of 1)` watermark only when the `min()` advances. This
holds the watermark back until every branch has drained.
- Producer-stamped branch identity. The `(i of N)` the WatermarkManager
needs is
stamped upstream, by the producing `ExecutableStage` when its output feeds
a
Flatten (a translation pre-pass records which PCollections are Flatten
inputs),
because Kafka Streams does not tell a processor which parent forwarded a
record.
Discussed with je-ik on Slack -- stamping at the producer is free since
Beam
fuses it and the runner (not user code) generates the watermark; a
single-input
consumer that happens to see an `(i, N)` stamp just ignores it.
- `FlattenTranslator` wires the N parents to one node; `FLATTEN` registered
in the
translator map.
Scope: only `ExecutableStage` producers stamp the branch identity so far,
which
covers PAssert's `GroupGlobally` (its Flatten inputs are stages). A `Read` /
`Impulse` / `GBK` output feeding a Flatten directly still reports `(0 of 1)`
--
that needs the same stamp wiring and is a follow-up when those tests are
enabled.
Test: `FlattenTest` unions two `Create -> ParDo` branches and asserts a
downstream
ParDo sees all four elements -- which only holds because the watermark is
held
until both branches drain.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
