On Wed, Apr 25, 2018 at 5:45 PM Xinyu Liu <[email protected]> wrote:
> Hi, > I am working on adding the stateful ParDo to the upcoming BEAM Samza runner, and realized that the state for each ParDo processElement() is not only associated with the window of the element, but also the key of the element. Chatted with Kenneth over email about this design decision, which has the following benefits for keyed state: > 1) No synchronization > 2) Simple programming model > 3) No communication between works > The current design doesn't support accessing the state across different keys, which seems to be a more general use case. This use case is also very common inside LinkedIn where the users have access to the entire state of an operator/task, and performing lookups and computations on top of it. It's quite hard to make every user here aware that the state is also tightly associated with key of the element.. Would side inputs be applicable here? (They're read-only, but other than that seem to fit the need.) > From the stateful ParDo API the state looks pretty general too. I am wondering is it possible to extend the current API to support both keyed and non-keyed state? Even internally BEAM assigns a dummy key for to associate the state with all the elements. It will be very beneficial to existing Samza users and help them adopt BEAM. Could you clarify how you would use this dummy key? You could manually add a random key, but in that case it's unlikely that any state stored would get observed again. Across what scope were you thinking state would be stored? The lifetime of the bundle? The worker? The job? How are conflicting writes resolved? Perhaps rather than describing the mechanism (state) that you're trying to use, it'd be helpful to describe the kinds of computations you're trying to perform, to figure out how the model should be adapted/extended if it doesn't meet those needs.
