junaiddshaukat opened a new pull request, #39380:
URL: https://github.com/apache/beam/pull/39380
Adds the \`validatesRunner\` gradle task, running Beam's @ValidatesRunner
tests
(from the SDK core test jar) against \`TestKafkaStreamsRunner\` -- category
excludes for features the runner does not support yet (side inputs, state,
timers, TestStream, unbounded, SDF, ...) plus a sickbay list for
known-failing
tests. Part of #18479; this is the milestone the midterm pointed at.
Current results:
- \`CreateTest\`: 5/5
- \`FlattenTest\`: 13/13, 1 sickbayed (needs multi-output executable stages
--
output-tag dispatch in the stage processor; follow-up)
Enabling the suite immediately caught three real bugs, fixed here:
1. Impulse bootstrap topics are now per-transform. A pipeline can contain two
Impulses (an empty \`Create\` plus the dummy branch \`PAssert\` adds
internally), and Kafka Streams rejects registering the same topic on two
source nodes.
2. \`Flatten\` is no longer declared in \`knownUrns()\`. Declaring it native
made
\`TrivialNativeTransformExpander\` corrupt the fused graph of a
multi-branch
Flatten whose inputs come from composite expansions (the fuser's own
validation failed with "consumed but never produced"). This one change
fixed
four failing tests. Mirrors the Flink runner, which keeps Read out of its
known URNs for expander reasons.
3. A Flatten of zero PCollections is rewritten before fusion into a primitive
Read of a new \`EmptyBoundedSource\`. A zero-input Flatten is a root of
the
pipeline graph, and \`GreedyPipelineFuser\` only accepts Impulse or Read
roots; an empty read has exactly the right semantics (no elements, then
the
terminal watermark) and reuses the existing Read translator.
Next: enabling the GroupByKey and ParDo suites and triaging their failures
the
same way, and a follow-up issue for multi-output executable stages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
