An update: the reference DirectRunner implementation of (and common execution code for) the Python user state and timers API has been merged: https://github.com/apache/beam/pull/6304
On Thu, Aug 30, 2018 at 1:48 AM Charles Chen <[email protected]> wrote: > Another update: the reference DirectRunner implementation of the Python > user state and timers API is out for review: > https://github.com/apache/beam/pull/6304 > > On Mon, Jul 9, 2018 at 2:18 PM Charles Chen <[email protected]> wrote: > >> An update: https://github.com/apache/beam/pull/5691 has been merged. I >> hope to send out a reference implementation in the DirectRunner soon. On >> the roadmap after that is work on the relevant portability interfaces here >> so we can get this working on runners like Beam Python on Flink. >> >> On Wed, Jun 20, 2018 at 10:00 AM Charles Chen <[email protected]> wrote: >> >>> An update on the implementation: I recently sent out the user-facing >>> pipeline construction part of the API implementation out for review: >>> https://github.com/apache/beam/pull/5691. >>> >>> On Tue, Jun 5, 2018 at 5:26 PM Charles Chen <[email protected]> wrote: >>> >>>> Thanks everyone for contributing here. We've reached rough consensus >>>> on the approach we should take with this API, and I've summarized this in >>>> the new "Community consensus" sections I added to the doc ( >>>> https://s.apache.org/beam-python-user-state-and-timers). I will begin >>>> initial implementation of this API soon. >>>> >>>> On Wed, May 23, 2018 at 8:08 PM Thomas Weise <[email protected]> wrote: >>>> >>>>> Nice proposal; it's exciting to see this about to be added to the SDK >>>>> as it enables a set of more complex use cases. >>>>> >>>>> I also think that some of the content can later be repurposed as user >>>>> documentation. >>>>> >>>>> Thanks, >>>>> Thomas >>>>> >>>>> >>>>> On Wed, May 23, 2018 at 11:49 AM, Charles Chen <[email protected]> wrote: >>>>> >>>>>> Thanks everyone for the detailed comments and discussions. It looks >>>>>> like by now, we mostly agree with the requirements and overall direction >>>>>> needed for the API, though there is continuing discussion on specific >>>>>> details. I want to highlight two new sections of the doc, which address >>>>>> some discussions that have come up: >>>>>> >>>>>> - *Existing state and transactionality*: this section addresses >>>>>> how we will address an existing transactionality inconsistency in the >>>>>> existing Java API. ( >>>>>> >>>>>> https://docs.google.com/document/d/1GadEkAmtbJQjmqiqfSzGw3b66TKerm8tyn6TK4blAys/edit#heading=h.ofyl9jspiz3b >>>>>> ) >>>>>> - *State for merging windows*: this section addresses how we will >>>>>> deal with non-combinable state in conjunction with merging windows. ( >>>>>> >>>>>> https://docs.google.com/document/d/1GadEkAmtbJQjmqiqfSzGw3b66TKerm8tyn6TK4blAys/edit#heading=h.ctxkcgabtzpy >>>>>> ) >>>>>> >>>>>> Let me know any further comments and suggestions. >>>>>> >>>>>> On Tue, May 22, 2018 at 9:29 AM Kenneth Knowles <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Nice. I know that Java users have found it helpful to have this >>>>>>> lower-level way of writing pipelines when the high-level primitives >>>>>>> don't >>>>>>> quite have the tight control they are looking for. I hope it will be a >>>>>>> big >>>>>>> draw for Python, too. >>>>>>> >>>>>>> (commenting on the doc) >>>>>>> >>>>>>> Kenn >>>>>>> >>>>>>> On Mon, May 21, 2018 at 5:15 PM Charles Chen <[email protected]> wrote: >>>>>>> >>>>>>>> I want to share a proposal for adding user state and timer support >>>>>>>> to the Beam Python SDK and get the community's thoughts on how such an >>>>>>>> API >>>>>>>> should look: https://s.apache.org/beam-python-user-state-and-timers >>>>>>>> >>>>>>>> Let me know what you think and please add any comments and >>>>>>>> suggestions you may have. >>>>>>>> >>>>>>>> Best, >>>>>>>> Charles >>>>>>>> >>>>>>> >>>>>
