Re: Create External Transform with WindowFn

Also it's strange that Java used (beam:window_fn:serialized_java:v1) for the URN here instead of "beam:window_fn:fixed_windows:v1" [1] which is what is being registered by Python [2]. This seems to be the immediate issue. Tracking bug for supporting custom windows is https://issues.apache.org/jira/

Re: Create External Transform with WindowFn

That is consistent with the stacktrace. The error is occurring in the from_runner_api call in dataflow_runner.py [1] [1] https://github.com/apache/beam/blob/e1ec7f63e40f50e4101e2f5339738545bfea7f9b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L492 On Tue, Aug 25, 2020 at 6:07 PM Ch

Re: Create External Transform with WindowFn

Pipelines that use external WindowingStrategies might be failing during proto -> object -> proto conversion we do today. This limitation will go away once Dataflow directly starts reading Beam protos. We are working on this now. Thanks, Cham On Tue, Aug 25, 2020 at 5:38 PM Boyuan Zhang wrote: >

Re: Create External Transform with WindowFn

Thanks, Robert! I want to add more details on my External PTransform: MyExternalPTransform -- expand to -- ParDo() -> WindowInto(FixWindow) -> ParDo() -> output void |

Re: @StateId uniqueness across DoFn(s)

Thank you all for the reply. One last question, I noticed that ParDoTest$StateTests > testValueStateSameId seems to be testing exact thi

Re: Create External Transform with WindowFn

You should be able to use a WindowInto with any of the common windowing operations (e.g. global, fixed, sliding, sessions) in an external transform. You should also be able to window into an arbitrary WindowFn as long as it produces standards window types, but if there's a bug here you could possib

Create External Transform with WindowFn

Hi team, I'm trying to create an External transform in Java SDK, which expands into several ParDo and a Window.into(FixWindow). When I use this transform in Python SDK, I get an pipeline construction error: apache_beam/utils/urns.py", line 186, in from_runner_api parameter_type, constructor =

Re: [External] Re: Memory Issue When Running Beam On Flink

I agree that this probably solves the described issue in the most straightforward way, but special handling for global window feels weird, as there is really nothing special about global window wrt state cleanup. Why is special handling for the global window weird? After all, it is a special ca

Re: JIRA - can't set resolution?

Yeah this is still broken as described in https://lists.apache.org/thread.html/r68924a0317a75d7858914b914e1d95fe36e0a9bf1794ef6861df7118%40%3Cdev.beam.apache.org%3E Currently blocked on https://issues.apache.org/jira/browse/INFRA-20563 On Tue, Aug 25, 2020 at 10:34 AM Udi Meiri wrote: > Example

JIRA - can't set resolution?

Example: https://issues.apache.org/jira/browse/BEAM-10751 When I click "resolve issue" the status changes to "resolved" but resolution is still "unresolved" and I can't change it. I believe there was a change a while back to JIRA? smime.p7s Description: S/MIME Cryptographic Signature