Hi Indika, I have started a basic implementation to test the concurrency handling, message order and transaction support of Redis. Basic message producers and consumers were implemented and messages are stored in a basic Redis queue. GitHub : https://github.com/Wishmitha/RedisTesting
Best Regards, On Thu, Jul 13, 2017 at 6:51 PM, Indika Sampath <indi...@wso2.com> wrote: > Hi Wishmitha, > > No SQL database is not an option as we have prior experience with > Cassandra. Basically, the important part is to find out performance and > fault tolerance capability in the database. Message broker > read/write/delete records all the time and database should be able to > handle high concurrency. Some metadata and message content write as byte > data. Also, message order matters when delivery. Hence database should > capable to sort records rather than bringing that logic into broker code. > As you mentioned transaction is another factor we are looking in the > database. Shall we start POC with Redis and see how it goes? > > Cheers! > > On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis <wishmi...@wso2.com> > wrote: > >> Hi all, >> >> I am working on a project to replace the current RDBMS based database of >> the message broker store with a file based database system. I have been >> researching on potential file based databases which can be used in the >> project scenario. I have evaluated the pros and cons of the each of the >> potential databases mainly based on read and write performances, >> transaction support, data structure (queue) implementation support and >> overhead of replacement. >> >> A summary of database evaluation is represented in the following table. >> >> >> Database >> >> Pros >> >> Cons >> >> LevelDB >> >> - >> >> A simple key value storage >> - >> >> Used in ActiveMQ (replaced with KahaDB) >> >> >> 1. >> >> Fast due to sorted keys >> 2. >> >> Simple transaction support with batch( ) operation >> 3. >> >> Sublevel feature (provides more organization and transactions >> facilitation) >> 4. >> >> LiveStream feature (able to query things which are yet being >> receiving) >> 5. >> >> Support triggers, groupby (mapreduce) and join (fanout) etc. >> >> >> 1. >> >> Performance decrease in concurrent / multi threaded scenarios. >> (improved in RocksDB.) >> 2. >> >> No object relational mapping. >> 3. >> >> Poor performance when memory is not enough. >> >> MongoDB >> >> >> - >> >> Document oriented >> - >> >> Can define JSON schemas >> >> >> >> 1. >> >> Direct object mapping (good replacement for RDBMS) >> 2. >> >> Can implement queue like data structures (store messages in separate >> documents and delete them once delivered / already implemented data >> structures : https://www.npmjs.com/package/mongodb-queue) >> 3. >> >> Capped collections (automatically remove old docs to make space for >> new docs.) >> >> >> 1. >> >> No transaction support for multiple documents. >> 2. >> >> Low performance compared to other NoSQL databases. >> >> Caassandra >> >> >> - >> >> Column oriented (a key is denoted by row and an attribute is denoted >> by a column which can dynamically change.) >> >> >> - >> >> Node - Cluster configuration (all nodes are identical) >> >> >> 1. >> >> High availability (No single point of failure. In case of node >> failure other identical nodes are available.) >> 2. >> >> No data loss due to replication (identical nodes) >> 3. >> >> Easy to model (column store is very similar to RDBMS tables.) >> 4. >> >> Similar query language to SQL (CQL) >> 5. >> >> High performance (No master node concept. Any node can provide >> requested query. No performance bottleneck.) >> 6. >> >> Transaction implementation >> >> >> 1. >> >> Does not support queue structure (message queue implementation is an >> anti-pattern : http://www.datastax.com/dev/bl >> og/cassandra-anti-patterns-queues-and-queue-like-datasets >> >> <http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets> >> ) >> >> Redis >> >> >> - >> >> A simple key value storage >> - >> >> In memory database >> - >> >> Support data structures like lists, caped lists, hashes, sets etc. ( >> https://redis.io/topics/data-types-intro >> <https://redis.io/topics/data-types-intro>) >> >> >> 1. >> >> Fast read,write performances >> 2. >> >> Provide persistence by writing to hard disk in configurable >> intervals. (snapshots : https://redis.io/topics/persistence) >> 3. >> >> Can implement message queue structures. (http://fiznool.com/blog/2016/ >> 02/24/building-a-simple-message-queue-with-redis/ >> >> <http://fiznool.com/blog/2016/02/24/building-a-simple-message-queue-with-redis/> >> ) >> 4. >> >> Use to implement message stores. (https://redis.io/topics/pubsub) >> 5. >> >> In built basic transaction support (MULTI/EXEC commands : >> https://redis.io/topics/transactions >> <https://redis.io/topics/transactions>) >> >> >> 1. >> >> Loss of data (in case of power outage etc.) >> 2. >> >> Depends on CPU performances when data amount is too high ( >> https://redis.io/topics/persistence >> <https://redis.io/topics/persistence>) >> >> RocksDB >> >> >> - >> >> A simple key value storage (upgrade of LevelDB) >> - >> >> In memory database >> >> >> 1. >> >> Fast read,write performances (faster than LevelDB) >> 2. >> >> Can implement queue structures (https://github.com/facebook/r >> ocksdb/wiki/Implement-Queue-Service-Using-RocksDB >> >> <https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB> >> ) >> 3. >> >> Support concurrency >> 4. >> >> Highly configurable (Pluggable Architecture) >> 5. >> >> Support persistence (memtables and transactional logs manage in >> memory. Memtable is flushed to a sstfile to provide persistence.) >> >> >> 1. >> >> Loss of data (in case of power outage etc.) >> 2. >> >> No in built transaction support (have to use TansactionDB : >> https://github.com/facebook/rocksdb/wiki/Transactions >> <https://github.com/facebook/rocksdb/wiki/Transactions> ) >> >> >> According to this evaluation I suggest Redis as the most suitable >> database for the message broker store. Even though it has an element of >> risk in data loss, the persistence storing is configurable unlike in other >> key-value stores. And it provides fast read and write performances compared >> to other databases with basic transaction support. >> >> And this is basically my evaluation and the opinion. It would be great to >> have a feedback on this, specially regarding the additional criteria to >> consider and other suggested databases. >> >> Best Regards >> >> -- >> >> *Wishmitha Mendis* >> >> *Intern - Software Engineering* >> *WSO2* >> >> *Mobile : +94 777577706 <+94%2077%20757%207706>* >> >> >> >> _______________________________________________ >> Architecture mailing list >> Architecture@wso2.org >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > Indika Sampath > Associate Technical Lead > WSO2 Inc. > http://wso2.com > > Phone: +94 716 424 744 <+94%2071%20642%204744> > Blog: http://indikasampath.blogspot.com/ > > > _______________________________________________ > Architecture mailing list > Architecture@wso2.org > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- *Wishmitha Mendis* *Intern - Software Engineering* *WSO2* *Mobile : +94 777577706*
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
