An update: https://github.com/apache/beam/pull/5691 has been merged. I hope to send out a reference implementation in the DirectRunner soon. On the roadmap after that is work on the relevant portability interfaces here so we can get this working on runners like Beam Python on Flink.
On Wed, Jun 20, 2018 at 10:00 AM Charles Chen <[email protected]> wrote: > An update on the implementation: I recently sent out the user-facing > pipeline construction part of the API implementation out for review: > https://github.com/apache/beam/pull/5691. > > On Tue, Jun 5, 2018 at 5:26 PM Charles Chen <[email protected]> wrote: > >> Thanks everyone for contributing here. We've reached rough consensus on >> the approach we should take with this API, and I've summarized this in the >> new "Community consensus" sections I added to the doc ( >> https://s.apache.org/beam-python-user-state-and-timers). I will begin >> initial implementation of this API soon. >> >> On Wed, May 23, 2018 at 8:08 PM Thomas Weise <[email protected]> wrote: >> >>> Nice proposal; it's exciting to see this about to be added to the SDK as >>> it enables a set of more complex use cases. >>> >>> I also think that some of the content can later be repurposed as user >>> documentation. >>> >>> Thanks, >>> Thomas >>> >>> >>> On Wed, May 23, 2018 at 11:49 AM, Charles Chen <[email protected]> wrote: >>> >>>> Thanks everyone for the detailed comments and discussions. It looks >>>> like by now, we mostly agree with the requirements and overall direction >>>> needed for the API, though there is continuing discussion on specific >>>> details. I want to highlight two new sections of the doc, which address >>>> some discussions that have come up: >>>> >>>> - *Existing state and transactionality*: this section addresses how >>>> we will address an existing transactionality inconsistency in the >>>> existing >>>> Java API. ( >>>> >>>> https://docs.google.com/document/d/1GadEkAmtbJQjmqiqfSzGw3b66TKerm8tyn6TK4blAys/edit#heading=h.ofyl9jspiz3b >>>> ) >>>> - *State for merging windows*: this section addresses how we will >>>> deal with non-combinable state in conjunction with merging windows. ( >>>> >>>> https://docs.google.com/document/d/1GadEkAmtbJQjmqiqfSzGw3b66TKerm8tyn6TK4blAys/edit#heading=h.ctxkcgabtzpy >>>> ) >>>> >>>> Let me know any further comments and suggestions. >>>> >>>> On Tue, May 22, 2018 at 9:29 AM Kenneth Knowles <[email protected]> wrote: >>>> >>>>> Nice. I know that Java users have found it helpful to have this >>>>> lower-level way of writing pipelines when the high-level primitives don't >>>>> quite have the tight control they are looking for. I hope it will be a big >>>>> draw for Python, too. >>>>> >>>>> (commenting on the doc) >>>>> >>>>> Kenn >>>>> >>>>> On Mon, May 21, 2018 at 5:15 PM Charles Chen <[email protected]> wrote: >>>>> >>>>>> I want to share a proposal for adding user state and timer support to >>>>>> the Beam Python SDK and get the community's thoughts on how such an API >>>>>> should look: https://s.apache.org/beam-python-user-state-and-timers >>>>>> >>>>>> Let me know what you think and please add any comments and >>>>>> suggestions you may have. >>>>>> >>>>>> Best, >>>>>> Charles >>>>>> >>>>> >>>
