GitHub user aglinxinyuan added a comment to the discussion: Why State flows with Data instead of being Centralized by the controller?
The reason we want to avoid global state in a pipelined distributed workflow system is probably similar to why we avoid global variables in a multithreaded program. We don't want one thread to modify a variable while another thread is using it. The same principle applies to the workflow. We don't want a downstream operator to modify some shared state while an upstream operator is still using it. We had a similar issue with large binaries. You can look at the previous discussions to see why we need locking for large binaries. The same concept is also covered in CS223, Transaction Processing and Distributed Data Management. GitHub link: https://github.com/apache/texera/discussions/7429#discussioncomment-17949468 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
