Hi All, Sandesh and I have been working on https://malhar.atlassian.net/browse/APEX-119 which aims to bring true distributed data structures to Apex. We have proposed a way to implement distributed data structures natively within Apache Apex. We have finished an analysis of what it would take to use other (non native) distributed data structures to accomplish the same task.
It turns out that using a distributed data structure fault tolerantly (like those provided by Apache Ignite) when it is not embedded into an Apex application is very difficult. It requires extra state and logic to manage windowing in a system outside of Apex, It also requires partitions reading from that data structure to communicate with one another anyway. The full analysis is here: https://docs.google.com/document/d/1Rau76WxAycyN9vQqP2bqDWZAwLw0u23xSh0_5fQ1980/edit?usp=sharing So the good news is if we want distributed data structures to be efficiently used in an Apex application, we'll need to implement distributed operators :) Thanks, Tim
