Hi, I'm new to Akka and CRDT, so take the below with a grain of salt. I'm trying to implement the application requirements with thousands of LWW-Maps, not related to each other. The implementation itself is not the main issue I have, but the internal "distributed data" gossip protocol implementation, I found interesting...
I suppose the problem below is related to my understanding (or lack of understanding) of CRDT, or current implementation of distributed-data plugin... Let's say I have 10K not related to each other CRDT based data type instances, for example 10K LWW-Maps. From the application point of view, they may be out of sync in relation to each other, but each one should be eventually consistent and synced in a cluster, using Local Consistency. By reading distributed-data plugin source code, I've found that for LWW-Map Key created (as any other CRDT data type instance defined) "lives" in a Replicator.dataEntries map. Then, the gossip protocol, based on the tick interval, replicates once a time configured chunk size of data, but no more than 10 chunks at once(Replicator.gossipTo()?). Means, the whole dataEntries map is replicated, no matter how many LWW-Maps were changed, if at all. The replication is goes as sending the digests of all (!) LWW-Maps (entries in dataEntries, using Status message) by gossip initiating node, and then requesting the differences and sending them back, if such differences were found (Gossip, one more round trip). That means, even with zero changes to my 10K LWW-Map entries, with few entries per each LWW-Map instance, 10K digests will be sent to all nodes each gossip tick fired. I see this behavior in a log... This is the flow seen in 2.4.16 release version. In 2.5.0-SNAPSHOT, delta CRDT approach is presented. Although the documentation and the article mentioned talk about deltas changes of different CRDT data types (ex.: c, d elements in a set), the replication mechanism works on key->dataEnvelope level, means on the whole changed entry of dataEntries map (deltaPropagationSelector.update(key.id, d) ...). Please notice, the replication is not performed as a sending delta changes of a single data type instance(changed entries of of particular Set for example), but as a whole Set replication. In the current implementation, "delta" means that only changed instances will be replicated by gossip protocol, and not the actual delta of CRDT data type instance. I spotted this behavior when having 1K different non related LWW-Maps with different key names, the system initiated sending of 1K digest signatures to all nodes without single change (in 2.4.16), or, having single map entry changed will fire the same flow anyway. 2.5.0-SNAPSHOT version should improve this approach, but, yet, the delta message is not built on a single CRDT type instance, but the all entries from dataEntires map there were changed, even if a single entry was changed in a 10 entries LWW-Map. So based on my understanding, my questions are: - in 2.4.16, what is the rationale to treat non related CRDT instances in dataEntries map as one composite CRDT, and replicate them as one single structure, and not as non-delta replication, but for each instance separately? If my approach is wrong, what is the way to do it anyway? - in 2.5.0, why the delta mechanism works on dataEntries map level, but the documentation and the article talk about delta of a single CRDT data type instance (members of CRDT Set) ? Is my understanding correct? If so, what is the reason for the current implementation? - What is the best way to implement large amount of non-related CRDT data type instances, Local Consistency, and reduce the replication overhead? Current implementation creates huge amount of inter-node traffic and cpu overhead, and even for less than 100K entries is pretty heavy. I don't really understand how the "shopping cart" like example (in case of Local Consistency!) with simultaneously logged-in 10K users with 10K shopping carts replicated each gossip interval (digest round trip saves a lot, but still) should work in real life. On my opinion, working with Local Consistency should improve performance in comparison with any other consistency strategy, but it looks like for large amount of CRDT data type instances, it's not the case in my performance tests ... P.S. I also found that "Subscribe" mechanism uses a kind of "delta" approach even in 2.4.16, as it saves all changed entries in "Replicator.changed" set for later notifications. So, the delta mechanism per entries of dataEntries map level exists, but only for notifications... Vadim Punski -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
