Inline
On Sat, Jan 20, 2024, 9:15 AM Jan Lukavský wrote:
> On 1/19/24 22:49, Robert Bradshaw via dev wrote:
>
> I think this standard design could still be made to work.
> Specifically, the graph would contain a DoFn that has the
> RequiresTimeSortedInput bit set, and as a single "subtransform"
On 1/19/24 22:49, Robert Bradshaw via dev wrote:
I think this standard design could still be made to work.
Specifically, the graph would contain a DoFn that has the
RequiresTimeSortedInput bit set, and as a single "subtransform" that
has a different DoFn in its spec that does not require this
I think this standard design could still be made to work.
Specifically, the graph would contain a DoFn that has the
RequiresTimeSortedInput bit set, and as a single "subtransform" that
has a different DoFn in its spec that does not require this bit to be
set and whose implementation enforces this
In this design space, what we have done in the past is:
1) ensure that runners all reject pipelines they cannot run correctly
2) if there is a default/workaround/slower implementation, provide it as an
override
This is largely ignoring portability but I think/hope it will still work.
At one time
I certainly don't have the deeper java insight here. So one more portable
based reply and then I'll step back on the Java specifics.
Portable runners only really have the "unknown Composite" fallback option,
where if the Composite's URN isn't known to the runner, it should use the
subgraph that
I was primarily focused on Java SDK (and core-contruction-java), but
generally speaking, any SDK can provide default expansion that runners
can use so that it is not (should not be) required to implement this
manually.
Currently, in Java SDK, the annotation is wired up into
StatefulDoFnRunner,
I agree that variable support across Runners does limit the adoption of a
feature. But it's also then limited if the SDKs and their local / direct
runners don't yet support the feature. The Go SDK doesn't currently have a way
of specifying that annotation, preventing use. (The lack of mention
Hi,
recently I came across the fact that most runners do not support
@RequiresTimeSortedInput annotation for sorting per-key data by event
timestamp [1]. Actually, runners supporting it seem to be Direct java,
Flink and Dataflow batch (as it is a noop there). The annotation has
use-cases in
