Daniel Halperin created BEAM-47:
-----------------------------------
Summary: UnboundedSource should support dynamic work rebalancing
Key: BEAM-47
URL: https://issues.apache.org/jira/browse/BEAM-47
Project: Beam
Issue Type: New Feature
Components: sdk-java-extensions
Reporter: Daniel Halperin
BoundedSource has splitAtFraction and related code to enable large work items
to be split dynamically and moved to other workers.
UnboundedSource does not have this support -- instead, runners are expected to
allocate many splits initially and rebalance those between workers.
We should add dynamic work rebalancing to UnboundedSource. This will almost
certainly be done by checkpointing the source, and then the dynamically
produced splits will both inherit the checkpoint (or a split-adapted version of
it).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
