Created related feature request https://github.com/apache/beam/issues/29789
We have to put more thought into exactly how to come up with merged
environments that do not result in conflicts. I prefer trying to
automatically do this on the SDK side instead of pushing the complexity to
the user (for
Yeah, we already have `ResourceHint.get_merged_value(cls, outer_value,
inner_value)` for reconciling resources within a composite, in the future
we could possibly just have another similar method and have the environment
merging logic hook into that.
On Fri, Dec 15, 2023 at 3:53 PM Robert
Hi Asmita,
Those both make sense to me, feel free to go ahead.
I'll be happy to review your PR when its ready
On Thu, Dec 14, 2023 at 11:44 AM Asmita Mutgekar
wrote:
> Hi Team,
>
> I have picked Issue: Add documentation and improved errors for QueryFn in
> MongoDbIO #21005
> Did some initial
There is definitely a body of future work in intelligently merging
compatible-but-not-equal environments. (Dataflow does this for example.)
Defining/detecting compatibility is not always easy, but sometimes is, and
we should at least cover those cases and grow them over time.
On Fri, Dec 15, 2023
Yeah I can confirm for the python runners (based on my reading of the
translations.py [1]) that only identical environments are merged together.
The funny thing is that we _originally_ implemented this hint as an
annotation but then changed it to hint because it semantically felt more
correct. I
That would do it. We got so tunnel visioned on side inputs we missed that!
IIRC the python local runner and Prism both only fuse transforms in
identical environments together. So any environmental diffs will prevent
fusion.
Runners as a rule are usually free to ignore/manage hints as they like.
I figured out my issue. I thought side inputs were breaking up my pipeline
but after experimenting with my transforms I now realize what was actually
breaking it up was different transform environments that weren't considered
compatible.
We have a custom resource hint (for specifying whether a
This is your daily summary of Beam's current high priority issues that may need
attention.
See https://beam.apache.org/contribute/issue-priorities for the meaning and
expectations around issue priorities.
Unassigned P1 Issues:
https://github.com/apache/beam/issues/29413 [Bug]: Can not use
