GitHub user carloea2 added a comment to the discussion: Why State flows with Data instead of being Centralized by the controller?
I’m not sure the global-variable analogy applies to the current design. As I understand it, once state is emitted into the workflow, operators do not share and mutate the same state object. Instead, each operator receives a state message and may forward a transformed version of it. That avoids concurrent mutation, but it does not eliminate state-management problems. It shifts them into message ordering, duplicated versions, stale reads, and deciding which transformed state is authoritative. A centralized design would not necessarily mean exposing one mutable object to every operator. The controller could maintain an immutable or versioned snapshot, allow operators to read it without locking, and serialize only the relatively rare updates. If most operators only read the state, lock contention should be limited. If, as I currently understand, only operators such as Loop End modify it, then the write path may be narrow enough to manage explicitly. In fact, centralizing writes could provide stronger consistency because there would be one place responsible for validating updates, ordering them, and publishing the latest version. By contrast, when state travels through the dataflow, multiple transformed copies can exist simultaneously, and correctness depends on the workflow topology and message ordering. So I think the key question is not simply “global state versus no global state.” It is: What concrete property does carrying state through the dataflow provide that a centralized, versioned, read-mostly state service would not? For example, is the goal to improve replayability, fault recovery, scalability, or operator isolation? Those would be stronger reasons than the shared-variable analogy alone. Please correct me if other operators currently modify the state, but my understanding is that Loop End is the main transformation point. GitHub link: https://github.com/apache/texera/discussions/7429#discussioncomment-17949819 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
