Re: [Architecture] File Based Database for Message Broker Store

[Manoj Gunawardena]

Hi All,

Are we use, Redis distributed setup?
In a MB containerized Cluster (DOCKER), how can we ensure, the data remain
when containers kill and start?
MB is supporting kubernates cluster?

Thanks

On Sun, Jul 16, 2017 at 9:41 AM, Wishmitha Mendis 
wrote:

> Hi Kevin,
>
> I just tried jedis for the initial implementation, no actual reason reason
> to be honest. I will try other java-clients you mentioned, and figure out
> what will fit the best for the project scenario.
>
> Best Regards,
>
> On Fri, Jul 14, 2017 at 11:17 PM, Kevin Ratnasekera 
> wrote:
>
>> Hi Wishmitha,
>>
>> There are many java clients available for redis, there are several very
>> good clients, see comparison [1]. Any reason using jedis over others?
>>
>> [1] https://redis.io/clients#java
>>
>> Regards
>> Kevin
>>
>>
>>
>> On Fri, Jul 14, 2017 at 11:02 PM, Wishmitha Mendis 
>> wrote:
>>
>>> Hi Indika,
>>>
>>> I have started a basic implementation to test the concurrency handling,
>>> message order and transaction support of Redis. Basic message producers and
>>> consumers were implemented and messages are stored in a basic Redis queue.
>>> GitHub : https://github.com/Wishmitha/RedisTesting
>>>
>>> Best Regards,
>>>
>>> On Thu, Jul 13, 2017 at 6:51 PM, Indika Sampath 
>>> wrote:
>>>
 Hi Wishmitha,

 No SQL database is not an option as we have prior experience with
 Cassandra. Basically, the important part is to find out performance and
 fault tolerance capability in the database. Message broker
 read/write/delete records all the time and database should be able to
 handle high concurrency. Some metadata and message content write as byte
 data. Also, message order matters when delivery. Hence database should
 capable to sort records rather than bringing that logic into broker code.
 As you mentioned transaction is another factor we are looking in the
 database. Shall we start POC with Redis and see how it goes?

 Cheers!

 On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis 
 wrote:

> Hi all,
>
> I am working on a project to replace the current RDBMS based database
> of the message broker store with a file based database system. I have been
> researching on potential file based databases which can be used in the
> project scenario. I have evaluated the pros and cons of the each of the
> potential databases mainly based on read and write performances,
> transaction support, data structure (queue) implementation support and
> overhead of replacement.
>
> A summary of database evaluation is represented in the following table.
>
>
> Database
>
> Pros
>
> Cons
>
> LevelDB
>
>-
>
>A simple key value storage
>-
>
>Used in ActiveMQ (replaced with KahaDB)
>
>
>1.
>
>Fast due to sorted keys
>2.
>
>Simple transaction support with batch( ) operation
>3.
>
>Sublevel feature (provides more organization and transactions
>facilitation)
>4.
>
>LiveStream feature (able to query things which are yet being
>receiving)
>5.
>
>Support triggers, groupby (mapreduce) and join (fanout) etc.
>
>
>1.
>
>Performance decrease in concurrent / multi threaded scenarios.
>(improved in RocksDB.)
>2.
>
>No object relational mapping.
>3.
>
>Poor performance when memory is not enough.
>
> MongoDB
>
>
>-
>
>Document oriented
>-
>
>Can define JSON schemas
>
>
>
>1.
>
>Direct object mapping (good replacement for RDBMS)
>2.
>
>Can implement queue like data structures (store messages in
>separate documents and delete them once delivered / already implemented
>data structures : https://www.npmjs.com/package/mongodb-queue)
>3.
>
>Capped collections (automatically remove old docs to make space
>for new docs.)
>
>
>1.
>
>No transaction support for multiple documents.
>2.
>
>Low performance compared to other NoSQL databases.
>
> Caassandra
>
>
>-
>
>Column oriented (a key is denoted by row and an attribute is
>denoted by a column which can dynamically change.)
>
>
>-
>
>Node - Cluster configuration (all nodes are identical)
>
>
>1.
>
>High availability (No single point of failure. In case of node
>failure other identical nodes are available.)
>2.
>
>No data loss due to replication (identical nodes)
>3.
>
>Easy to model (column store is very similar to RDBMS tables.)
>4.
>
>Similar query language to SQL (CQL)
>5.
>
>High 

Re: [Architecture] File Based Database for Message Broker Store

[Wishmitha Mendis]

Hi Kevin,

I just tried jedis for the initial implementation, no actual reason reason
to be honest. I will try other java-clients you mentioned, and figure out
what will fit the best for the project scenario.

Best Regards,

On Fri, Jul 14, 2017 at 11:17 PM, Kevin Ratnasekera  wrote:

> Hi Wishmitha,
>
> There are many java clients available for redis, there are several very
> good clients, see comparison [1]. Any reason using jedis over others?
>
> [1] https://redis.io/clients#java
>
> Regards
> Kevin
>
>
>
> On Fri, Jul 14, 2017 at 11:02 PM, Wishmitha Mendis 
> wrote:
>
>> Hi Indika,
>>
>> I have started a basic implementation to test the concurrency handling,
>> message order and transaction support of Redis. Basic message producers and
>> consumers were implemented and messages are stored in a basic Redis queue.
>> GitHub : https://github.com/Wishmitha/RedisTesting
>>
>> Best Regards,
>>
>> On Thu, Jul 13, 2017 at 6:51 PM, Indika Sampath  wrote:
>>
>>> Hi Wishmitha,
>>>
>>> No SQL database is not an option as we have prior experience with
>>> Cassandra. Basically, the important part is to find out performance and
>>> fault tolerance capability in the database. Message broker
>>> read/write/delete records all the time and database should be able to
>>> handle high concurrency. Some metadata and message content write as byte
>>> data. Also, message order matters when delivery. Hence database should
>>> capable to sort records rather than bringing that logic into broker code.
>>> As you mentioned transaction is another factor we are looking in the
>>> database. Shall we start POC with Redis and see how it goes?
>>>
>>> Cheers!
>>>
>>> On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis 
>>> wrote:
>>>
 Hi all,

 I am working on a project to replace the current RDBMS based database
 of the message broker store with a file based database system. I have been
 researching on potential file based databases which can be used in the
 project scenario. I have evaluated the pros and cons of the each of the
 potential databases mainly based on read and write performances,
 transaction support, data structure (queue) implementation support and
 overhead of replacement.

 A summary of database evaluation is represented in the following table.


 Database

 Pros

 Cons

 LevelDB

-

A simple key value storage
-

Used in ActiveMQ (replaced with KahaDB)


1.

Fast due to sorted keys
2.

Simple transaction support with batch( ) operation
3.

Sublevel feature (provides more organization and transactions
facilitation)
4.

LiveStream feature (able to query things which are yet being
receiving)
5.

Support triggers, groupby (mapreduce) and join (fanout) etc.


1.

Performance decrease in concurrent / multi threaded scenarios.
(improved in RocksDB.)
2.

No object relational mapping.
3.

Poor performance when memory is not enough.

 MongoDB


-

Document oriented
-

Can define JSON schemas



1.

Direct object mapping (good replacement for RDBMS)
2.

Can implement queue like data structures (store messages in
separate documents and delete them once delivered / already implemented
data structures : https://www.npmjs.com/package/mongodb-queue)
3.

Capped collections (automatically remove old docs to make space for
new docs.)


1.

No transaction support for multiple documents.
2.

Low performance compared to other NoSQL databases.

 Caassandra


-

Column oriented (a key is denoted by row and an attribute is
denoted by a column which can dynamically change.)


-

Node - Cluster configuration (all nodes are identical)


1.

High availability (No single point of failure. In case of node
failure other identical nodes are available.)
2.

No data loss due to replication (identical nodes)
3.

Easy to model (column store is very similar to RDBMS tables.)
4.

Similar query language to SQL (CQL)
5.

High performance (No master node concept. Any node can provide
requested query. No performance bottleneck.)
6.

Transaction implementation


1.

Does not support queue structure (message queue implementation is
an anti-pattern : http://www.datastax.com/dev/bl
og/cassandra-anti-patterns-queues-and-queue-like-datasets

 

Re: [Architecture] File Based Database for Message Broker Store

[Manuri Amaya Perera]

On Fri, Jul 14, 2017 at 11:38 AM, Asanka Abeyweera 
wrote:

> Hi Manuri,
>
> Rather than syncing periodically (1-second or less), they say they can
> sync at every query with AOF [1]. I don't' know the performance impact of
> that approach. Need to test and see. I am not proposing to use Redis. Just
> pointing the information.
>
​
Right.. Thanks for the information.
​I missed what had been said about AOF before replying.

>
> [1] https://redis.io/topics/persistence
>
> On Fri, Jul 14, 2017 at 10:27 AM, Manuri Amaya Perera 
> wrote:
>
>> Hi,
>>
>> From the given description, it seems persistence is supported by writing
>> to hard disk in configurable intervals. Unless we have a very low interval
>> (which might be against any expected performance improvement brought by the
>> database being in-memory) it cannot be assured that there won't be data
>> loss. Also even a low interval would not fully guarantee that.
>> I had a quick look at [1] and it says "You can configure Redis to have it
>> save the dataset every N seconds if there are at least M changes in the
>> dataset, or you can manually call the SAVE or BGSAVE commands.". So, if
>> server crashes we could lose 1 - M changes. We can only reduce it but
>> cannot avoid it completely. :)
>> So I'm not sure if we could use Redis.
>>
>> [1] https://redis.io/topics/persistence
>>
>> Thanks,
>> Manuri
>>
>> On Fri, Jul 14, 2017 at 9:39 AM, Asanka Abeyweera 
>> wrote:
>>
>>> Hi Wishmitha,
>>>
>>> If we use AOF wth fsync at every query, will there be a data loss with
>>> Redis? AOF might not have very good performance compared to RDB. But as
>>> Sasikal mentioned we cannot allow message losses in the broker.
>>>
>>> On Fri, Jul 14, 2017 at 8:42 AM, Riyafa Abdul Hameed 
>>> wrote:
>>>
 Dear Wishmitha,

 For all the databases mentioned please provide references to the
 relevant websites and where you got the information from. You have
 mentioned for some which is great and you should do so for others as well.
 Also when referencing don't have the weblink inline with the statement
 because it distracts the viewer as the links might be long. You can number
 the references and refer them below (You don't need to follow any
 standards, simply giving the link would suffice).

 Cheers,
 Riyafa

 On Fri, Jul 14, 2017 at 7:48 AM, Sasikala Kottegoda 
 wrote:

> Hi Wishmitha,
>
> We cannot tolerate data loss in the system. Guaranteed delivery comes
> first, performance comes next for the broker.
>
> Thank you,
> Sasikala
>
> On Fri, Jul 14, 2017 at 6:48 AM, Malaka Gangananda 
> wrote:
>
>> Hi All,
>>
>> I have attached the pdf of above mentioned document.
>>
>> Thanks.
>>
>> On Fri, Jul 14, 2017 at 6:38 AM, Malaka Gangananda 
>> wrote:
>>
>>> Hi Wishmitha,
>>>
>>> I have done some research work about existing message brokers
>>> previously, and here I have attach the link to the document which I have
>>> recorded the details about ActiveMQ Artemis high performance file 
>>> journal
>>> and other file based message broker systems [1]. I think it will be 
>>> useful
>>> to get an idea about how and why existing message brokers have 
>>> implemented
>>> file based message stores.
>>>
>>> [1] https://docs.google.com/a/wso2.com/document/d/19CKLoAIZrue5O
>>> 047AnNOL6E24S4KcbNhWF81Mb3Id4E/edit?usp=sharing
>>>
>>> Thanks,
>>>
>>> On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis <
>>> wishmi...@wso2.com> wrote:
>>>
 Hi all,

 I am working on a project to replace the current RDBMS based
 database of the message broker store with a file based database 
 system. I
 have been researching on potential file based databases which can be 
 used
 in the project scenario. I have evaluated the pros and cons of the 
 each of
 the potential databases mainly based on read and write performances,
 transaction support, data structure (queue) implementation support and
 overhead of replacement.

 A summary of database evaluation is represented in the following
 table.


 Database

 Pros

 Cons

 LevelDB

-

A simple key value storage
-

Used in ActiveMQ (replaced with KahaDB)


1.

Fast due to sorted keys
2.

Simple transaction support with batch( ) operation
3.

Sublevel feature (provides more organization and transactions
facilitation)
4.

LiveStream feature (able to query things which are yet being
receiving)
5.

Suppor

Re: [Architecture] File Based Database for Message Broker Store

[Wishmitha Mendis]

Hi  Maryam,

Even the recommended "appendfsync everysec" mode has a potential data loss,
the only option would be trying out fsync at every query. So it would be
useful to implement a POC as you suggested based on "appendfsync always" and
test the performances whether it can deliver the desired results.

Best Regards,

On Fri, Jul 14, 2017 at 10:27 AM, Manuri Amaya Perera 
wrote:

> Hi,
>
> From the given description, it seems persistence is supported by writing
> to hard disk in configurable intervals. Unless we have a very low interval
> (which might be against any expected performance improvement brought by the
> database being in-memory) it cannot be assured that there won't be data
> loss. Also even a low interval would not fully guarantee that.
> I had a quick look at [1] and it says "You can configure Redis to have it
> save the dataset every N seconds if there are at least M changes in the
> dataset, or you can manually call the SAVE or BGSAVE commands.". So, if
> server crashes we could lose 1 - M changes. We can only reduce it but
> cannot avoid it completely. :)
> So I'm not sure if we could use Redis.
>
> [1] https://redis.io/topics/persistence
>
> Thanks,
> Manuri
>
> On Fri, Jul 14, 2017 at 9:39 AM, Asanka Abeyweera 
> wrote:
>
>> Hi Wishmitha,
>>
>> If we use AOF wth fsync at every query, will there be a data loss with
>> Redis? AOF might not have very good performance compared to RDB. But as
>> Sasikal mentioned we cannot allow message losses in the broker.
>>
>> On Fri, Jul 14, 2017 at 8:42 AM, Riyafa Abdul Hameed 
>> wrote:
>>
>>> Dear Wishmitha,
>>>
>>> For all the databases mentioned please provide references to the
>>> relevant websites and where you got the information from. You have
>>> mentioned for some which is great and you should do so for others as well.
>>> Also when referencing don't have the weblink inline with the statement
>>> because it distracts the viewer as the links might be long. You can number
>>> the references and refer them below (You don't need to follow any
>>> standards, simply giving the link would suffice).
>>>
>>> Cheers,
>>> Riyafa
>>>
>>> On Fri, Jul 14, 2017 at 7:48 AM, Sasikala Kottegoda 
>>> wrote:
>>>
 Hi Wishmitha,

 We cannot tolerate data loss in the system. Guaranteed delivery comes
 first, performance comes next for the broker.

 Thank you,
 Sasikala

 On Fri, Jul 14, 2017 at 6:48 AM, Malaka Gangananda 
 wrote:

> Hi All,
>
> I have attached the pdf of above mentioned document.
>
> Thanks.
>
> On Fri, Jul 14, 2017 at 6:38 AM, Malaka Gangananda 
> wrote:
>
>> Hi Wishmitha,
>>
>> I have done some research work about existing message brokers
>> previously, and here I have attach the link to the document which I have
>> recorded the details about ActiveMQ Artemis high performance file journal
>> and other file based message broker systems [1]. I think it will be 
>> useful
>> to get an idea about how and why existing message brokers have 
>> implemented
>> file based message stores.
>>
>> [1] https://docs.google.com/a/wso2.com/document/d/19CKLoAIZrue5O
>> 047AnNOL6E24S4KcbNhWF81Mb3Id4E/edit?usp=sharing
>>
>> Thanks,
>>
>> On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis > > wrote:
>>
>>> Hi all,
>>>
>>> I am working on a project to replace the current RDBMS based
>>> database of the message broker store with a file based database system. 
>>> I
>>> have been researching on potential file based databases which can be 
>>> used
>>> in the project scenario. I have evaluated the pros and cons of the each 
>>> of
>>> the potential databases mainly based on read and write performances,
>>> transaction support, data structure (queue) implementation support and
>>> overhead of replacement.
>>>
>>> A summary of database evaluation is represented in the following
>>> table.
>>>
>>>
>>> Database
>>>
>>> Pros
>>>
>>> Cons
>>>
>>> LevelDB
>>>
>>>-
>>>
>>>A simple key value storage
>>>-
>>>
>>>Used in ActiveMQ (replaced with KahaDB)
>>>
>>>
>>>1.
>>>
>>>Fast due to sorted keys
>>>2.
>>>
>>>Simple transaction support with batch( ) operation
>>>3.
>>>
>>>Sublevel feature (provides more organization and transactions
>>>facilitation)
>>>4.
>>>
>>>LiveStream feature (able to query things which are yet being
>>>receiving)
>>>5.
>>>
>>>Support triggers, groupby (mapreduce) and join (fanout) etc.
>>>
>>>
>>>1.
>>>
>>>Performance decrease in concurrent / multi threaded scenarios.
>>>(improved in RocksDB.)
>>>2.
>>>
>>>No object relational mapping.
>>>3.
>>>
>>>Poor performance when mem

Re: [Architecture] File Based Database for Message Broker Store

[Kevin Ratnasekera]

Hi Wishmitha,

There are many java clients available for redis, there are several very
good clients, see comparison [1]. Any reason using jedis over others?

[1] https://redis.io/clients#java

Regards
Kevin



On Fri, Jul 14, 2017 at 11:02 PM, Wishmitha Mendis 
wrote:

> Hi Indika,
>
> I have started a basic implementation to test the concurrency handling,
> message order and transaction support of Redis. Basic message producers and
> consumers were implemented and messages are stored in a basic Redis queue.
> GitHub : https://github.com/Wishmitha/RedisTesting
>
> Best Regards,
>
> On Thu, Jul 13, 2017 at 6:51 PM, Indika Sampath  wrote:
>
>> Hi Wishmitha,
>>
>> No SQL database is not an option as we have prior experience with
>> Cassandra. Basically, the important part is to find out performance and
>> fault tolerance capability in the database. Message broker
>> read/write/delete records all the time and database should be able to
>> handle high concurrency. Some metadata and message content write as byte
>> data. Also, message order matters when delivery. Hence database should
>> capable to sort records rather than bringing that logic into broker code.
>> As you mentioned transaction is another factor we are looking in the
>> database. Shall we start POC with Redis and see how it goes?
>>
>> Cheers!
>>
>> On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis 
>> wrote:
>>
>>> Hi all,
>>>
>>> I am working on a project to replace the current RDBMS based database of
>>> the message broker store with a file based database system. I have been
>>> researching on potential file based databases which can be used in the
>>> project scenario. I have evaluated the pros and cons of the each of the
>>> potential databases mainly based on read and write performances,
>>> transaction support, data structure (queue) implementation support and
>>> overhead of replacement.
>>>
>>> A summary of database evaluation is represented in the following table.
>>>
>>>
>>> Database
>>>
>>> Pros
>>>
>>> Cons
>>>
>>> LevelDB
>>>
>>>-
>>>
>>>A simple key value storage
>>>-
>>>
>>>Used in ActiveMQ (replaced with KahaDB)
>>>
>>>
>>>1.
>>>
>>>Fast due to sorted keys
>>>2.
>>>
>>>Simple transaction support with batch( ) operation
>>>3.
>>>
>>>Sublevel feature (provides more organization and transactions
>>>facilitation)
>>>4.
>>>
>>>LiveStream feature (able to query things which are yet being
>>>receiving)
>>>5.
>>>
>>>Support triggers, groupby (mapreduce) and join (fanout) etc.
>>>
>>>
>>>1.
>>>
>>>Performance decrease in concurrent / multi threaded scenarios.
>>>(improved in RocksDB.)
>>>2.
>>>
>>>No object relational mapping.
>>>3.
>>>
>>>Poor performance when memory is not enough.
>>>
>>> MongoDB
>>>
>>>
>>>-
>>>
>>>Document oriented
>>>-
>>>
>>>Can define JSON schemas
>>>
>>>
>>>
>>>1.
>>>
>>>Direct object mapping (good replacement for RDBMS)
>>>2.
>>>
>>>Can implement queue like data structures (store messages in separate
>>>documents and delete them once delivered / already implemented data
>>>structures : https://www.npmjs.com/package/mongodb-queue)
>>>3.
>>>
>>>Capped collections (automatically remove old docs to make space for
>>>new docs.)
>>>
>>>
>>>1.
>>>
>>>No transaction support for multiple documents.
>>>2.
>>>
>>>Low performance compared to other NoSQL databases.
>>>
>>> Caassandra
>>>
>>>
>>>-
>>>
>>>Column oriented (a key is denoted by row and an attribute is denoted
>>>by a column which can dynamically change.)
>>>
>>>
>>>-
>>>
>>>Node - Cluster configuration (all nodes are identical)
>>>
>>>
>>>1.
>>>
>>>High availability (No single point of failure. In case of node
>>>failure other identical nodes are available.)
>>>2.
>>>
>>>No data loss due to replication (identical nodes)
>>>3.
>>>
>>>Easy to model (column store is very similar to RDBMS tables.)
>>>4.
>>>
>>>Similar query language to SQL (CQL)
>>>5.
>>>
>>>High performance (No master node concept. Any node can provide
>>>requested query. No performance bottleneck.)
>>>6.
>>>
>>>Transaction implementation
>>>
>>>
>>>1.
>>>
>>>Does not support queue structure (message queue implementation is an
>>>anti-pattern : http://www.datastax.com/dev/bl
>>>og/cassandra-anti-patterns-queues-and-queue-like-datasets
>>>
>>> 
>>>)
>>>
>>> Redis
>>>
>>>
>>>-
>>>
>>>A simple key value storage
>>>-
>>>
>>>In memory database
>>>-
>>>
>>>Support data structures like lists, caped lists, hashes, sets etc. (
>>>https://redis.io/topics/data-types-intro
>>>)
>>>
>>>
>>>1.
>>>
>>>Fast read,write performances
>>>2.
>>>
>>>Provide persistence by writing to hard disk 

Re: [Architecture] File Based Database for Message Broker Store

[Wishmitha Mendis]

Hi Indika,

I have started a basic implementation to test the concurrency handling,
message order and transaction support of Redis. Basic message producers and
consumers were implemented and messages are stored in a basic Redis queue.
GitHub : https://github.com/Wishmitha/RedisTesting

Best Regards,

On Thu, Jul 13, 2017 at 6:51 PM, Indika Sampath  wrote:

> Hi Wishmitha,
>
> No SQL database is not an option as we have prior experience with
> Cassandra. Basically, the important part is to find out performance and
> fault tolerance capability in the database. Message broker
> read/write/delete records all the time and database should be able to
> handle high concurrency. Some metadata and message content write as byte
> data. Also, message order matters when delivery. Hence database should
> capable to sort records rather than bringing that logic into broker code.
> As you mentioned transaction is another factor we are looking in the
> database. Shall we start POC with Redis and see how it goes?
>
> Cheers!
>
> On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis 
> wrote:
>
>> Hi all,
>>
>> I am working on a project to replace the current RDBMS based database of
>> the message broker store with a file based database system. I have been
>> researching on potential file based databases which can be used in the
>> project scenario. I have evaluated the pros and cons of the each of the
>> potential databases mainly based on read and write performances,
>> transaction support, data structure (queue) implementation support and
>> overhead of replacement.
>>
>> A summary of database evaluation is represented in the following table.
>>
>>
>> Database
>>
>> Pros
>>
>> Cons
>>
>> LevelDB
>>
>>-
>>
>>A simple key value storage
>>-
>>
>>Used in ActiveMQ (replaced with KahaDB)
>>
>>
>>1.
>>
>>Fast due to sorted keys
>>2.
>>
>>Simple transaction support with batch( ) operation
>>3.
>>
>>Sublevel feature (provides more organization and transactions
>>facilitation)
>>4.
>>
>>LiveStream feature (able to query things which are yet being
>>receiving)
>>5.
>>
>>Support triggers, groupby (mapreduce) and join (fanout) etc.
>>
>>
>>1.
>>
>>Performance decrease in concurrent / multi threaded scenarios.
>>(improved in RocksDB.)
>>2.
>>
>>No object relational mapping.
>>3.
>>
>>Poor performance when memory is not enough.
>>
>> MongoDB
>>
>>
>>-
>>
>>Document oriented
>>-
>>
>>Can define JSON schemas
>>
>>
>>
>>1.
>>
>>Direct object mapping (good replacement for RDBMS)
>>2.
>>
>>Can implement queue like data structures (store messages in separate
>>documents and delete them once delivered / already implemented data
>>structures : https://www.npmjs.com/package/mongodb-queue)
>>3.
>>
>>Capped collections (automatically remove old docs to make space for
>>new docs.)
>>
>>
>>1.
>>
>>No transaction support for multiple documents.
>>2.
>>
>>Low performance compared to other NoSQL databases.
>>
>> Caassandra
>>
>>
>>-
>>
>>Column oriented (a key is denoted by row and an attribute is denoted
>>by a column which can dynamically change.)
>>
>>
>>-
>>
>>Node - Cluster configuration (all nodes are identical)
>>
>>
>>1.
>>
>>High availability (No single point of failure. In case of node
>>failure other identical nodes are available.)
>>2.
>>
>>No data loss due to replication (identical nodes)
>>3.
>>
>>Easy to model (column store is very similar to RDBMS tables.)
>>4.
>>
>>Similar query language to SQL (CQL)
>>5.
>>
>>High performance (No master node concept. Any node can provide
>>requested query. No performance bottleneck.)
>>6.
>>
>>Transaction implementation
>>
>>
>>1.
>>
>>Does not support queue structure (message queue implementation is an
>>anti-pattern : http://www.datastax.com/dev/bl
>>og/cassandra-anti-patterns-queues-and-queue-like-datasets
>>
>> 
>>)
>>
>> Redis
>>
>>
>>-
>>
>>A simple key value storage
>>-
>>
>>In memory database
>>-
>>
>>Support data structures like lists, caped lists, hashes, sets etc. (
>>https://redis.io/topics/data-types-intro
>>)
>>
>>
>>1.
>>
>>Fast read,write performances
>>2.
>>
>>Provide persistence by writing to hard disk in configurable
>>intervals. (snapshots : https://redis.io/topics/persistence)
>>3.
>>
>>Can implement message queue structures. (http://fiznool.com/blog/2016/
>>02/24/building-a-simple-message-queue-with-redis/
>>
>> 
>>)
>>4.
>>
>>Use to implement message stores. (https://redis.io/topics/pubsub)
>>5.
>>
>>In built basic transaction support 

Re: [Architecture] File Based Database for Message Broker Store

[Pamod Sylvester]

For file based journals/key-value stores transactions should be handled
explicitly.

Let's also evaluate how other brokering technologies i.e artemis, kafka
uses file based systems as well. That will be a good comparison to narrow
down the list. Also we could evaluate how MariaDB has used levelDB as a
store as well, since MariaDB uses LevelDB as it's storage engine [1].

Artemis supports features like Tx,Dx

[1] https://mariadb.com/kb/en/mariadb/leveldb/

Thanks,
Pamod

On Fri, Jul 14, 2017 at 4:25 PM, Asanka Abeyweera  wrote:

> Hi Wishmitha,
>
> Since Redis is not an RDBMS it is interesting to know how the transactions
> are handled. When implementing distributed transactions feature we required
> row level locking instead of table level locking. We need to check if Redis
> also supports row level locking kind of feature. Further, in the article
> [1], it says  Redis does not support rollbacks. Need to investigate more
> and see if this is a blocker for MB.
>
> [1] https://redis.io/topics/transactions
>
> On Fri, Jul 14, 2017 at 3:21 PM, Maryam Ziyad  wrote:
>
>> Hi Wishmitha, All,
>>
>> It would be interesting to also look at how AOF persistence in Redis and
>> the append only journal used by Apache ActiveMQ Artemis [1] compare.
>>
>> Regarding AOF, fsync and durability, this
>> 
>> blog post pointed at by the official Redis site, even though a bit old,
>> could be useful. It mentions "Data committed to the disk using the
>> fsync(2) system call (or equivalent) that gives us, *virtually*, data
>> safety against complete system failure like a power outage." as a safety
>> level [2]. Also, out of the three options made available for fsync
>> (appendfsync no, appendfsync everysec and appendfsync always) they mention
>> that "appendfsync always" (with every query) is the best in terms of
>> durability, but is obviously slower compared to the other modes, and
>> recommend "appendfsync everysec" as a good balance between durability and
>> speed [3].
>>
>> Since these configurations seem straightforward and Redis has other
>> useful advantages as mentioned, doing a POC with Redis and testing seems
>> like a suitable approach.
>>
>> [1] https://activemq.apache.org/artemis/docs/2.1.0/persistence.html
>> [2] http://oldblog.antirez.com/post/redis-persistence-demystified.html -
>> "What we can't control"
>> [3] http://oldblog.antirez.com/post/redis-persistence-demystified.html -
>> "AOF durability"
>>
>> Best Regards,
>> Maryam
>>
>> On Fri, Jul 14, 2017 at 11:38 AM, Asanka Abeyweera 
>> wrote:
>>
>>> Hi Manuri,
>>>
>>> Rather than syncing periodically (1-second or less), they say they can
>>> sync at every query with AOF [1]. I don't' know the performance impact
>>> of that approach. Need to test and see. I am not proposing to use Redis.
>>> Just pointing the information.
>>>
>>> [1] https://redis.io/topics/persistence
>>>
>>> On Fri, Jul 14, 2017 at 10:27 AM, Manuri Amaya Perera 
>>> wrote:
>>>
 Hi,

 From the given description, it seems persistence is supported by
 writing to hard disk in configurable intervals. Unless we have a very low
 interval (which might be against any expected performance improvement
 brought by the database being in-memory) it cannot be assured that there
 won't be data loss. Also even a low interval would not fully guarantee
 that.
 I had a quick look at [1] and it says "You can configure Redis to have
 it save the dataset every N seconds if there are at least M changes in the
 dataset, or you can manually call the SAVE or BGSAVE commands.". So, if
 server crashes we could lose 1 - M changes. We can only reduce it but
 cannot avoid it completely. :)
 So I'm not sure if we could use Redis.

 [1] https://redis.io/topics/persistence

 Thanks,
 Manuri

 On Fri, Jul 14, 2017 at 9:39 AM, Asanka Abeyweera 
 wrote:

> Hi Wishmitha,
>
> If we use AOF wth fsync at every query, will there be a data loss with
> Redis? AOF might not have very good performance compared to RDB. But as
> Sasikal mentioned we cannot allow message losses in the broker.
>
> On Fri, Jul 14, 2017 at 8:42 AM, Riyafa Abdul Hameed 
> wrote:
>
>> Dear Wishmitha,
>>
>> For all the databases mentioned please provide references to the
>> relevant websites and where you got the information from. You have
>> mentioned for some which is great and you should do so for others as 
>> well.
>> Also when referencing don't have the weblink inline with the statement
>> because it distracts the viewer as the links might be long. You can 
>> number
>> the references and refer them below (You don't need to follow any
>> standards, simply giving the link would suffice).
>>
>> Cheers,
>> Riyafa
>>
>> On Fri, Jul 14, 2017 at 7:48 AM, Sasikala Kottegoda <
>> sasik...@wso2.com> wrote:

Re: [Architecture] File Based Database for Message Broker Store

[Asanka Abeyweera]

Hi Wishmitha,

Since Redis is not an RDBMS it is interesting to know how the transactions
are handled. When implementing distributed transactions feature we required
row level locking instead of table level locking. We need to check if Redis
also supports row level locking kind of feature. Further, in the article
[1], it says  Redis does not support rollbacks. Need to investigate more
and see if this is a blocker for MB.

[1] https://redis.io/topics/transactions

On Fri, Jul 14, 2017 at 3:21 PM, Maryam Ziyad  wrote:

> Hi Wishmitha, All,
>
> It would be interesting to also look at how AOF persistence in Redis and
> the append only journal used by Apache ActiveMQ Artemis [1] compare.
>
> Regarding AOF, fsync and durability, this
>  blog
> post pointed at by the official Redis site, even though a bit old, could be
> useful. It mentions "Data committed to the disk using the fsync(2) system
> call (or equivalent) that gives us, *virtually*, data safety against
> complete system failure like a power outage." as a safety level [2].
> Also, out of the three options made available for fsync (appendfsync no,
> appendfsync everysec and appendfsync always) they mention that "appendfsync
> always" (with every query) is the best in terms of durability, but is
> obviously slower compared to the other modes, and recommend "appendfsync
> everysec" as a good balance between durability and speed [3].
>
> Since these configurations seem straightforward and Redis has other useful
> advantages as mentioned, doing a POC with Redis and testing seems like a
> suitable approach.
>
> [1] https://activemq.apache.org/artemis/docs/2.1.0/persistence.html
> [2] http://oldblog.antirez.com/post/redis-persistence-demystified.html -
> "What we can't control"
> [3] http://oldblog.antirez.com/post/redis-persistence-demystified.html -
> "AOF durability"
>
> Best Regards,
> Maryam
>
> On Fri, Jul 14, 2017 at 11:38 AM, Asanka Abeyweera 
> wrote:
>
>> Hi Manuri,
>>
>> Rather than syncing periodically (1-second or less), they say they can
>> sync at every query with AOF [1]. I don't' know the performance impact
>> of that approach. Need to test and see. I am not proposing to use Redis.
>> Just pointing the information.
>>
>> [1] https://redis.io/topics/persistence
>>
>> On Fri, Jul 14, 2017 at 10:27 AM, Manuri Amaya Perera 
>> wrote:
>>
>>> Hi,
>>>
>>> From the given description, it seems persistence is supported by writing
>>> to hard disk in configurable intervals. Unless we have a very low interval
>>> (which might be against any expected performance improvement brought by the
>>> database being in-memory) it cannot be assured that there won't be data
>>> loss. Also even a low interval would not fully guarantee that.
>>> I had a quick look at [1] and it says "You can configure Redis to have
>>> it save the dataset every N seconds if there are at least M changes in the
>>> dataset, or you can manually call the SAVE or BGSAVE commands.". So, if
>>> server crashes we could lose 1 - M changes. We can only reduce it but
>>> cannot avoid it completely. :)
>>> So I'm not sure if we could use Redis.
>>>
>>> [1] https://redis.io/topics/persistence
>>>
>>> Thanks,
>>> Manuri
>>>
>>> On Fri, Jul 14, 2017 at 9:39 AM, Asanka Abeyweera 
>>> wrote:
>>>
 Hi Wishmitha,

 If we use AOF wth fsync at every query, will there be a data loss with
 Redis? AOF might not have very good performance compared to RDB. But as
 Sasikal mentioned we cannot allow message losses in the broker.

 On Fri, Jul 14, 2017 at 8:42 AM, Riyafa Abdul Hameed 
 wrote:

> Dear Wishmitha,
>
> For all the databases mentioned please provide references to the
> relevant websites and where you got the information from. You have
> mentioned for some which is great and you should do so for others as well.
> Also when referencing don't have the weblink inline with the statement
> because it distracts the viewer as the links might be long. You can number
> the references and refer them below (You don't need to follow any
> standards, simply giving the link would suffice).
>
> Cheers,
> Riyafa
>
> On Fri, Jul 14, 2017 at 7:48 AM, Sasikala Kottegoda  > wrote:
>
>> Hi Wishmitha,
>>
>> We cannot tolerate data loss in the system. Guaranteed delivery comes
>> first, performance comes next for the broker.
>>
>> Thank you,
>> Sasikala
>>
>> On Fri, Jul 14, 2017 at 6:48 AM, Malaka Gangananda 
>> wrote:
>>
>>> Hi All,
>>>
>>> I have attached the pdf of above mentioned document.
>>>
>>> Thanks.
>>>
>>> On Fri, Jul 14, 2017 at 6:38 AM, Malaka Gangananda >> > wrote:
>>>
 Hi Wishmitha,

 I have done some research work about existing message brokers
 previously, and here I have attach the link to the document which I 
 have

Re: [Architecture] File Based Database for Message Broker Store

[Maryam Ziyad]

Hi Wishmitha, All,

It would be interesting to also look at how AOF persistence in Redis and
the append only journal used by Apache ActiveMQ Artemis [1] compare.

Regarding AOF, fsync and durability, this
 blog
post pointed at by the official Redis site, even though a bit old, could be
useful. It mentions "Data committed to the disk using the fsync(2) system
call (or equivalent) that gives us, *virtually*, data safety against
complete system failure like a power outage." as a safety level [2]. Also,
out of the three options made available for fsync (appendfsync no,
appendfsync everysec and appendfsync always) they mention that "appendfsync
always" (with every query) is the best in terms of durability, but is
obviously slower compared to the other modes, and recommend "appendfsync
everysec" as a good balance between durability and speed [3].

Since these configurations seem straightforward and Redis has other useful
advantages as mentioned, doing a POC with Redis and testing seems like a
suitable approach.

[1] https://activemq.apache.org/artemis/docs/2.1.0/persistence.html
[2] http://oldblog.antirez.com/post/redis-persistence-demystified.html -
"What we can't control"
[3] http://oldblog.antirez.com/post/redis-persistence-demystified.html -
"AOF durability"

Best Regards,
Maryam

On Fri, Jul 14, 2017 at 11:38 AM, Asanka Abeyweera 
wrote:

> Hi Manuri,
>
> Rather than syncing periodically (1-second or less), they say they can
> sync at every query with AOF [1]. I don't' know the performance impact of
> that approach. Need to test and see. I am not proposing to use Redis. Just
> pointing the information.
>
> [1] https://redis.io/topics/persistence
>
> On Fri, Jul 14, 2017 at 10:27 AM, Manuri Amaya Perera 
> wrote:
>
>> Hi,
>>
>> From the given description, it seems persistence is supported by writing
>> to hard disk in configurable intervals. Unless we have a very low interval
>> (which might be against any expected performance improvement brought by the
>> database being in-memory) it cannot be assured that there won't be data
>> loss. Also even a low interval would not fully guarantee that.
>> I had a quick look at [1] and it says "You can configure Redis to have it
>> save the dataset every N seconds if there are at least M changes in the
>> dataset, or you can manually call the SAVE or BGSAVE commands.". So, if
>> server crashes we could lose 1 - M changes. We can only reduce it but
>> cannot avoid it completely. :)
>> So I'm not sure if we could use Redis.
>>
>> [1] https://redis.io/topics/persistence
>>
>> Thanks,
>> Manuri
>>
>> On Fri, Jul 14, 2017 at 9:39 AM, Asanka Abeyweera 
>> wrote:
>>
>>> Hi Wishmitha,
>>>
>>> If we use AOF wth fsync at every query, will there be a data loss with
>>> Redis? AOF might not have very good performance compared to RDB. But as
>>> Sasikal mentioned we cannot allow message losses in the broker.
>>>
>>> On Fri, Jul 14, 2017 at 8:42 AM, Riyafa Abdul Hameed 
>>> wrote:
>>>
 Dear Wishmitha,

 For all the databases mentioned please provide references to the
 relevant websites and where you got the information from. You have
 mentioned for some which is great and you should do so for others as well.
 Also when referencing don't have the weblink inline with the statement
 because it distracts the viewer as the links might be long. You can number
 the references and refer them below (You don't need to follow any
 standards, simply giving the link would suffice).

 Cheers,
 Riyafa

 On Fri, Jul 14, 2017 at 7:48 AM, Sasikala Kottegoda 
 wrote:

> Hi Wishmitha,
>
> We cannot tolerate data loss in the system. Guaranteed delivery comes
> first, performance comes next for the broker.
>
> Thank you,
> Sasikala
>
> On Fri, Jul 14, 2017 at 6:48 AM, Malaka Gangananda 
> wrote:
>
>> Hi All,
>>
>> I have attached the pdf of above mentioned document.
>>
>> Thanks.
>>
>> On Fri, Jul 14, 2017 at 6:38 AM, Malaka Gangananda 
>> wrote:
>>
>>> Hi Wishmitha,
>>>
>>> I have done some research work about existing message brokers
>>> previously, and here I have attach the link to the document which I have
>>> recorded the details about ActiveMQ Artemis high performance file 
>>> journal
>>> and other file based message broker systems [1]. I think it will be 
>>> useful
>>> to get an idea about how and why existing message brokers have 
>>> implemented
>>> file based message stores.
>>>
>>> [1] https://docs.google.com/a/wso2.com/document/d/19CKLoAIZrue5O
>>> 047AnNOL6E24S4KcbNhWF81Mb3Id4E/edit?usp=sharing
>>>
>>> Thanks,
>>>
>>> On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis <
>>> wishmi...@wso2.com> wrote:
>>>
 Hi all,

 I am working on a project to replace the current RDBMS based
>

Re: [Architecture] File Based Database for Message Broker Store

[Asanka Abeyweera]

Hi Manuri,

Rather than syncing periodically (1-second or less), they say they can sync at
every query with AOF [1]. I don't' know the performance impact of that
approach. Need to test and see. I am not proposing to use Redis. Just
pointing the information.

[1] https://redis.io/topics/persistence

On Fri, Jul 14, 2017 at 10:27 AM, Manuri Amaya Perera 
wrote:

> Hi,
>
> From the given description, it seems persistence is supported by writing
> to hard disk in configurable intervals. Unless we have a very low interval
> (which might be against any expected performance improvement brought by the
> database being in-memory) it cannot be assured that there won't be data
> loss. Also even a low interval would not fully guarantee that.
> I had a quick look at [1] and it says "You can configure Redis to have it
> save the dataset every N seconds if there are at least M changes in the
> dataset, or you can manually call the SAVE or BGSAVE commands.". So, if
> server crashes we could lose 1 - M changes. We can only reduce it but
> cannot avoid it completely. :)
> So I'm not sure if we could use Redis.
>
> [1] https://redis.io/topics/persistence
>
> Thanks,
> Manuri
>
> On Fri, Jul 14, 2017 at 9:39 AM, Asanka Abeyweera 
> wrote:
>
>> Hi Wishmitha,
>>
>> If we use AOF wth fsync at every query, will there be a data loss with
>> Redis? AOF might not have very good performance compared to RDB. But as
>> Sasikal mentioned we cannot allow message losses in the broker.
>>
>> On Fri, Jul 14, 2017 at 8:42 AM, Riyafa Abdul Hameed 
>> wrote:
>>
>>> Dear Wishmitha,
>>>
>>> For all the databases mentioned please provide references to the
>>> relevant websites and where you got the information from. You have
>>> mentioned for some which is great and you should do so for others as well.
>>> Also when referencing don't have the weblink inline with the statement
>>> because it distracts the viewer as the links might be long. You can number
>>> the references and refer them below (You don't need to follow any
>>> standards, simply giving the link would suffice).
>>>
>>> Cheers,
>>> Riyafa
>>>
>>> On Fri, Jul 14, 2017 at 7:48 AM, Sasikala Kottegoda 
>>> wrote:
>>>
 Hi Wishmitha,

 We cannot tolerate data loss in the system. Guaranteed delivery comes
 first, performance comes next for the broker.

 Thank you,
 Sasikala

 On Fri, Jul 14, 2017 at 6:48 AM, Malaka Gangananda 
 wrote:

> Hi All,
>
> I have attached the pdf of above mentioned document.
>
> Thanks.
>
> On Fri, Jul 14, 2017 at 6:38 AM, Malaka Gangananda 
> wrote:
>
>> Hi Wishmitha,
>>
>> I have done some research work about existing message brokers
>> previously, and here I have attach the link to the document which I have
>> recorded the details about ActiveMQ Artemis high performance file journal
>> and other file based message broker systems [1]. I think it will be 
>> useful
>> to get an idea about how and why existing message brokers have 
>> implemented
>> file based message stores.
>>
>> [1] https://docs.google.com/a/wso2.com/document/d/19CKLoAIZrue5O
>> 047AnNOL6E24S4KcbNhWF81Mb3Id4E/edit?usp=sharing
>>
>> Thanks,
>>
>> On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis > > wrote:
>>
>>> Hi all,
>>>
>>> I am working on a project to replace the current RDBMS based
>>> database of the message broker store with a file based database system. 
>>> I
>>> have been researching on potential file based databases which can be 
>>> used
>>> in the project scenario. I have evaluated the pros and cons of the each 
>>> of
>>> the potential databases mainly based on read and write performances,
>>> transaction support, data structure (queue) implementation support and
>>> overhead of replacement.
>>>
>>> A summary of database evaluation is represented in the following
>>> table.
>>>
>>>
>>> Database
>>>
>>> Pros
>>>
>>> Cons
>>>
>>> LevelDB
>>>
>>>-
>>>
>>>A simple key value storage
>>>-
>>>
>>>Used in ActiveMQ (replaced with KahaDB)
>>>
>>>
>>>1.
>>>
>>>Fast due to sorted keys
>>>2.
>>>
>>>Simple transaction support with batch( ) operation
>>>3.
>>>
>>>Sublevel feature (provides more organization and transactions
>>>facilitation)
>>>4.
>>>
>>>LiveStream feature (able to query things which are yet being
>>>receiving)
>>>5.
>>>
>>>Support triggers, groupby (mapreduce) and join (fanout) etc.
>>>
>>>
>>>1.
>>>
>>>Performance decrease in concurrent / multi threaded scenarios.
>>>(improved in RocksDB.)
>>>2.
>>>
>>>No object relational mapping.
>>>3.
>>>
>>>Poor performance when memory is not enough.
>>

Re: [Architecture] File Based Database for Message Broker Store

[Manuri Amaya Perera]

Hi,

>From the given description, it seems persistence is supported by writing to
hard disk in configurable intervals. Unless we have a very low interval
(which might be against any expected performance improvement brought by the
database being in-memory) it cannot be assured that there won't be data
loss. Also even a low interval would not fully guarantee that.
I had a quick look at [1] and it says "You can configure Redis to have it
save the dataset every N seconds if there are at least M changes in the
dataset, or you can manually call the SAVE or BGSAVE commands.". So, if
server crashes we could lose 1 - M changes. We can only reduce it but
cannot avoid it completely. :)
So I'm not sure if we could use Redis.

[1] https://redis.io/topics/persistence

Thanks,
Manuri

On Fri, Jul 14, 2017 at 9:39 AM, Asanka Abeyweera  wrote:

> Hi Wishmitha,
>
> If we use AOF wth fsync at every query, will there be a data loss with
> Redis? AOF might not have very good performance compared to RDB. But as
> Sasikal mentioned we cannot allow message losses in the broker.
>
> On Fri, Jul 14, 2017 at 8:42 AM, Riyafa Abdul Hameed 
> wrote:
>
>> Dear Wishmitha,
>>
>> For all the databases mentioned please provide references to the relevant
>> websites and where you got the information from. You have mentioned for
>> some which is great and you should do so for others as well. Also when
>> referencing don't have the weblink inline with the statement because it
>> distracts the viewer as the links might be long. You can number the
>> references and refer them below (You don't need to follow any standards,
>> simply giving the link would suffice).
>>
>> Cheers,
>> Riyafa
>>
>> On Fri, Jul 14, 2017 at 7:48 AM, Sasikala Kottegoda 
>> wrote:
>>
>>> Hi Wishmitha,
>>>
>>> We cannot tolerate data loss in the system. Guaranteed delivery comes
>>> first, performance comes next for the broker.
>>>
>>> Thank you,
>>> Sasikala
>>>
>>> On Fri, Jul 14, 2017 at 6:48 AM, Malaka Gangananda 
>>> wrote:
>>>
 Hi All,

 I have attached the pdf of above mentioned document.

 Thanks.

 On Fri, Jul 14, 2017 at 6:38 AM, Malaka Gangananda 
 wrote:

> Hi Wishmitha,
>
> I have done some research work about existing message brokers
> previously, and here I have attach the link to the document which I have
> recorded the details about ActiveMQ Artemis high performance file journal
> and other file based message broker systems [1]. I think it will be useful
> to get an idea about how and why existing message brokers have implemented
> file based message stores.
>
> [1] https://docs.google.com/a/wso2.com/document/d/19CKLoAIZrue5O
> 047AnNOL6E24S4KcbNhWF81Mb3Id4E/edit?usp=sharing
>
> Thanks,
>
> On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis 
> wrote:
>
>> Hi all,
>>
>> I am working on a project to replace the current RDBMS based database
>> of the message broker store with a file based database system. I have 
>> been
>> researching on potential file based databases which can be used in the
>> project scenario. I have evaluated the pros and cons of the each of the
>> potential databases mainly based on read and write performances,
>> transaction support, data structure (queue) implementation support and
>> overhead of replacement.
>>
>> A summary of database evaluation is represented in the following
>> table.
>>
>>
>> Database
>>
>> Pros
>>
>> Cons
>>
>> LevelDB
>>
>>-
>>
>>A simple key value storage
>>-
>>
>>Used in ActiveMQ (replaced with KahaDB)
>>
>>
>>1.
>>
>>Fast due to sorted keys
>>2.
>>
>>Simple transaction support with batch( ) operation
>>3.
>>
>>Sublevel feature (provides more organization and transactions
>>facilitation)
>>4.
>>
>>LiveStream feature (able to query things which are yet being
>>receiving)
>>5.
>>
>>Support triggers, groupby (mapreduce) and join (fanout) etc.
>>
>>
>>1.
>>
>>Performance decrease in concurrent / multi threaded scenarios.
>>(improved in RocksDB.)
>>2.
>>
>>No object relational mapping.
>>3.
>>
>>Poor performance when memory is not enough.
>>
>> MongoDB
>>
>>
>>-
>>
>>Document oriented
>>-
>>
>>Can define JSON schemas
>>
>>
>>
>>1.
>>
>>Direct object mapping (good replacement for RDBMS)
>>2.
>>
>>Can implement queue like data structures (store messages in
>>separate documents and delete them once delivered / already 
>> implemented
>>data structures : https://www.npmjs.com/package/mongodb-queue)
>>3.
>>
>>Capped collections (automatically remo

Re: [Architecture] File Based Database for Message Broker Store

[Asanka Abeyweera]

Hi Wishmitha,

If we use AOF wth fsync at every query, will there be a data loss with
Redis? AOF might not have very good performance compared to RDB. But as
Sasikal mentioned we cannot allow message losses in the broker.

On Fri, Jul 14, 2017 at 8:42 AM, Riyafa Abdul Hameed 
wrote:

> Dear Wishmitha,
>
> For all the databases mentioned please provide references to the relevant
> websites and where you got the information from. You have mentioned for
> some which is great and you should do so for others as well. Also when
> referencing don't have the weblink inline with the statement because it
> distracts the viewer as the links might be long. You can number the
> references and refer them below (You don't need to follow any standards,
> simply giving the link would suffice).
>
> Cheers,
> Riyafa
>
> On Fri, Jul 14, 2017 at 7:48 AM, Sasikala Kottegoda 
> wrote:
>
>> Hi Wishmitha,
>>
>> We cannot tolerate data loss in the system. Guaranteed delivery comes
>> first, performance comes next for the broker.
>>
>> Thank you,
>> Sasikala
>>
>> On Fri, Jul 14, 2017 at 6:48 AM, Malaka Gangananda 
>> wrote:
>>
>>> Hi All,
>>>
>>> I have attached the pdf of above mentioned document.
>>>
>>> Thanks.
>>>
>>> On Fri, Jul 14, 2017 at 6:38 AM, Malaka Gangananda 
>>> wrote:
>>>
 Hi Wishmitha,

 I have done some research work about existing message brokers
 previously, and here I have attach the link to the document which I have
 recorded the details about ActiveMQ Artemis high performance file journal
 and other file based message broker systems [1]. I think it will be useful
 to get an idea about how and why existing message brokers have implemented
 file based message stores.

 [1] https://docs.google.com/a/wso2.com/document/d/19CKLoAIZrue5O
 047AnNOL6E24S4KcbNhWF81Mb3Id4E/edit?usp=sharing

 Thanks,

 On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis 
 wrote:

> Hi all,
>
> I am working on a project to replace the current RDBMS based database
> of the message broker store with a file based database system. I have been
> researching on potential file based databases which can be used in the
> project scenario. I have evaluated the pros and cons of the each of the
> potential databases mainly based on read and write performances,
> transaction support, data structure (queue) implementation support and
> overhead of replacement.
>
> A summary of database evaluation is represented in the following table.
>
>
> Database
>
> Pros
>
> Cons
>
> LevelDB
>
>-
>
>A simple key value storage
>-
>
>Used in ActiveMQ (replaced with KahaDB)
>
>
>1.
>
>Fast due to sorted keys
>2.
>
>Simple transaction support with batch( ) operation
>3.
>
>Sublevel feature (provides more organization and transactions
>facilitation)
>4.
>
>LiveStream feature (able to query things which are yet being
>receiving)
>5.
>
>Support triggers, groupby (mapreduce) and join (fanout) etc.
>
>
>1.
>
>Performance decrease in concurrent / multi threaded scenarios.
>(improved in RocksDB.)
>2.
>
>No object relational mapping.
>3.
>
>Poor performance when memory is not enough.
>
> MongoDB
>
>
>-
>
>Document oriented
>-
>
>Can define JSON schemas
>
>
>
>1.
>
>Direct object mapping (good replacement for RDBMS)
>2.
>
>Can implement queue like data structures (store messages in
>separate documents and delete them once delivered / already implemented
>data structures : https://www.npmjs.com/package/mongodb-queue)
>3.
>
>Capped collections (automatically remove old docs to make space
>for new docs.)
>
>
>1.
>
>No transaction support for multiple documents.
>2.
>
>Low performance compared to other NoSQL databases.
>
> Caassandra
>
>
>-
>
>Column oriented (a key is denoted by row and an attribute is
>denoted by a column which can dynamically change.)
>
>
>-
>
>Node - Cluster configuration (all nodes are identical)
>
>
>1.
>
>High availability (No single point of failure. In case of node
>failure other identical nodes are available.)
>2.
>
>No data loss due to replication (identical nodes)
>3.
>
>Easy to model (column store is very similar to RDBMS tables.)
>4.
>
>Similar query language to SQL (CQL)
>5.
>
>High performance (No master node concept. Any node can provide
>requested query. No performance bottlen

Re: [Architecture] File Based Database for Message Broker Store

[Riyafa Abdul Hameed]

Dear Wishmitha,

For all the databases mentioned please provide references to the relevant
websites and where you got the information from. You have mentioned for
some which is great and you should do so for others as well. Also when
referencing don't have the weblink inline with the statement because it
distracts the viewer as the links might be long. You can number the
references and refer them below (You don't need to follow any standards,
simply giving the link would suffice).

Cheers,
Riyafa

On Fri, Jul 14, 2017 at 7:48 AM, Sasikala Kottegoda 
wrote:

> Hi Wishmitha,
>
> We cannot tolerate data loss in the system. Guaranteed delivery comes
> first, performance comes next for the broker.
>
> Thank you,
> Sasikala
>
> On Fri, Jul 14, 2017 at 6:48 AM, Malaka Gangananda 
> wrote:
>
>> Hi All,
>>
>> I have attached the pdf of above mentioned document.
>>
>> Thanks.
>>
>> On Fri, Jul 14, 2017 at 6:38 AM, Malaka Gangananda 
>> wrote:
>>
>>> Hi Wishmitha,
>>>
>>> I have done some research work about existing message brokers
>>> previously, and here I have attach the link to the document which I have
>>> recorded the details about ActiveMQ Artemis high performance file journal
>>> and other file based message broker systems [1]. I think it will be useful
>>> to get an idea about how and why existing message brokers have implemented
>>> file based message stores.
>>>
>>> [1] https://docs.google.com/a/wso2.com/document/d/19CKLoAIZrue5O
>>> 047AnNOL6E24S4KcbNhWF81Mb3Id4E/edit?usp=sharing
>>>
>>> Thanks,
>>>
>>> On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis 
>>> wrote:
>>>
 Hi all,

 I am working on a project to replace the current RDBMS based database
 of the message broker store with a file based database system. I have been
 researching on potential file based databases which can be used in the
 project scenario. I have evaluated the pros and cons of the each of the
 potential databases mainly based on read and write performances,
 transaction support, data structure (queue) implementation support and
 overhead of replacement.

 A summary of database evaluation is represented in the following table.


 Database

 Pros

 Cons

 LevelDB

-

A simple key value storage
-

Used in ActiveMQ (replaced with KahaDB)


1.

Fast due to sorted keys
2.

Simple transaction support with batch( ) operation
3.

Sublevel feature (provides more organization and transactions
facilitation)
4.

LiveStream feature (able to query things which are yet being
receiving)
5.

Support triggers, groupby (mapreduce) and join (fanout) etc.


1.

Performance decrease in concurrent / multi threaded scenarios.
(improved in RocksDB.)
2.

No object relational mapping.
3.

Poor performance when memory is not enough.

 MongoDB


-

Document oriented
-

Can define JSON schemas



1.

Direct object mapping (good replacement for RDBMS)
2.

Can implement queue like data structures (store messages in
separate documents and delete them once delivered / already implemented
data structures : https://www.npmjs.com/package/mongodb-queue)
3.

Capped collections (automatically remove old docs to make space for
new docs.)


1.

No transaction support for multiple documents.
2.

Low performance compared to other NoSQL databases.

 Caassandra


-

Column oriented (a key is denoted by row and an attribute is
denoted by a column which can dynamically change.)


-

Node - Cluster configuration (all nodes are identical)


1.

High availability (No single point of failure. In case of node
failure other identical nodes are available.)
2.

No data loss due to replication (identical nodes)
3.

Easy to model (column store is very similar to RDBMS tables.)
4.

Similar query language to SQL (CQL)
5.

High performance (No master node concept. Any node can provide
requested query. No performance bottleneck.)
6.

Transaction implementation


1.

Does not support queue structure (message queue implementation is
an anti-pattern : http://www.datastax.com/dev/bl
og/cassandra-anti-patterns-queues-and-queue-like-datasets

 
)

 Redis


-

A simple key value storage
>>>

Re: [Architecture] File Based Database for Message Broker Store

[Sasikala Kottegoda]

Hi Wishmitha,

We cannot tolerate data loss in the system. Guaranteed delivery comes
first, performance comes next for the broker.

Thank you,
Sasikala

On Fri, Jul 14, 2017 at 6:48 AM, Malaka Gangananda  wrote:

> Hi All,
>
> I have attached the pdf of above mentioned document.
>
> Thanks.
>
> On Fri, Jul 14, 2017 at 6:38 AM, Malaka Gangananda 
> wrote:
>
>> Hi Wishmitha,
>>
>> I have done some research work about existing message brokers previously,
>> and here I have attach the link to the document which I have recorded the
>> details about ActiveMQ Artemis high performance file journal and other file
>> based message broker systems [1]. I think it will be useful to get an idea
>> about how and why existing message brokers have implemented file based
>> message stores.
>>
>> [1] https://docs.google.com/a/wso2.com/document/d/19CKLoAIZrue5O
>> 047AnNOL6E24S4KcbNhWF81Mb3Id4E/edit?usp=sharing
>>
>> Thanks,
>>
>> On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis 
>> wrote:
>>
>>> Hi all,
>>>
>>> I am working on a project to replace the current RDBMS based database of
>>> the message broker store with a file based database system. I have been
>>> researching on potential file based databases which can be used in the
>>> project scenario. I have evaluated the pros and cons of the each of the
>>> potential databases mainly based on read and write performances,
>>> transaction support, data structure (queue) implementation support and
>>> overhead of replacement.
>>>
>>> A summary of database evaluation is represented in the following table.
>>>
>>>
>>> Database
>>>
>>> Pros
>>>
>>> Cons
>>>
>>> LevelDB
>>>
>>>-
>>>
>>>A simple key value storage
>>>-
>>>
>>>Used in ActiveMQ (replaced with KahaDB)
>>>
>>>
>>>1.
>>>
>>>Fast due to sorted keys
>>>2.
>>>
>>>Simple transaction support with batch( ) operation
>>>3.
>>>
>>>Sublevel feature (provides more organization and transactions
>>>facilitation)
>>>4.
>>>
>>>LiveStream feature (able to query things which are yet being
>>>receiving)
>>>5.
>>>
>>>Support triggers, groupby (mapreduce) and join (fanout) etc.
>>>
>>>
>>>1.
>>>
>>>Performance decrease in concurrent / multi threaded scenarios.
>>>(improved in RocksDB.)
>>>2.
>>>
>>>No object relational mapping.
>>>3.
>>>
>>>Poor performance when memory is not enough.
>>>
>>> MongoDB
>>>
>>>
>>>-
>>>
>>>Document oriented
>>>-
>>>
>>>Can define JSON schemas
>>>
>>>
>>>
>>>1.
>>>
>>>Direct object mapping (good replacement for RDBMS)
>>>2.
>>>
>>>Can implement queue like data structures (store messages in separate
>>>documents and delete them once delivered / already implemented data
>>>structures : https://www.npmjs.com/package/mongodb-queue)
>>>3.
>>>
>>>Capped collections (automatically remove old docs to make space for
>>>new docs.)
>>>
>>>
>>>1.
>>>
>>>No transaction support for multiple documents.
>>>2.
>>>
>>>Low performance compared to other NoSQL databases.
>>>
>>> Caassandra
>>>
>>>
>>>-
>>>
>>>Column oriented (a key is denoted by row and an attribute is denoted
>>>by a column which can dynamically change.)
>>>
>>>
>>>-
>>>
>>>Node - Cluster configuration (all nodes are identical)
>>>
>>>
>>>1.
>>>
>>>High availability (No single point of failure. In case of node
>>>failure other identical nodes are available.)
>>>2.
>>>
>>>No data loss due to replication (identical nodes)
>>>3.
>>>
>>>Easy to model (column store is very similar to RDBMS tables.)
>>>4.
>>>
>>>Similar query language to SQL (CQL)
>>>5.
>>>
>>>High performance (No master node concept. Any node can provide
>>>requested query. No performance bottleneck.)
>>>6.
>>>
>>>Transaction implementation
>>>
>>>
>>>1.
>>>
>>>Does not support queue structure (message queue implementation is an
>>>anti-pattern : http://www.datastax.com/dev/bl
>>>og/cassandra-anti-patterns-queues-and-queue-like-datasets
>>>
>>> 
>>>)
>>>
>>> Redis
>>>
>>>
>>>-
>>>
>>>A simple key value storage
>>>-
>>>
>>>In memory database
>>>-
>>>
>>>Support data structures like lists, caped lists, hashes, sets etc. (
>>>https://redis.io/topics/data-types-intro
>>>)
>>>
>>>
>>>1.
>>>
>>>Fast read,write performances
>>>2.
>>>
>>>Provide persistence by writing to hard disk in configurable
>>>intervals. (snapshots : https://redis.io/topics/persistence)
>>>3.
>>>
>>>Can implement message queue structures. (
>>>http://fiznool.com/blog/2016/02/24/building-a-simple-messag
>>>e-queue-with-redis/
>>>
>>> 
>>>)
>>>4.
>>>
>>>Use to implement message stores. (https:

Re: [Architecture] File Based Database for Message Broker Store

[Malaka Gangananda]

Hi Wishmitha,

I have done some research work about existing message brokers previously,
and here I have attach the link to the document which I have recorded the
details about ActiveMQ Artemis high performance file journal and other file
based message broker systems [1]. I think it will be useful to get an idea
about how and why existing message brokers have implemented file based
message stores.

[1]
https://docs.google.com/a/wso2.com/document/d/19CKLoAIZrue5O047AnNOL6E24S4KcbNhWF81Mb3Id4E/edit?usp=sharing

Thanks,

On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis 
wrote:

> Hi all,
>
> I am working on a project to replace the current RDBMS based database of
> the message broker store with a file based database system. I have been
> researching on potential file based databases which can be used in the
> project scenario. I have evaluated the pros and cons of the each of the
> potential databases mainly based on read and write performances,
> transaction support, data structure (queue) implementation support and
> overhead of replacement.
>
> A summary of database evaluation is represented in the following table.
>
>
> Database
>
> Pros
>
> Cons
>
> LevelDB
>
>-
>
>A simple key value storage
>-
>
>Used in ActiveMQ (replaced with KahaDB)
>
>
>1.
>
>Fast due to sorted keys
>2.
>
>Simple transaction support with batch( ) operation
>3.
>
>Sublevel feature (provides more organization and transactions
>facilitation)
>4.
>
>LiveStream feature (able to query things which are yet being receiving)
>5.
>
>Support triggers, groupby (mapreduce) and join (fanout) etc.
>
>
>1.
>
>Performance decrease in concurrent / multi threaded scenarios.
>(improved in RocksDB.)
>2.
>
>No object relational mapping.
>3.
>
>Poor performance when memory is not enough.
>
> MongoDB
>
>
>-
>
>Document oriented
>-
>
>Can define JSON schemas
>
>
>
>1.
>
>Direct object mapping (good replacement for RDBMS)
>2.
>
>Can implement queue like data structures (store messages in separate
>documents and delete them once delivered / already implemented data
>structures : https://www.npmjs.com/package/mongodb-queue)
>3.
>
>Capped collections (automatically remove old docs to make space for
>new docs.)
>
>
>1.
>
>No transaction support for multiple documents.
>2.
>
>Low performance compared to other NoSQL databases.
>
> Caassandra
>
>
>-
>
>Column oriented (a key is denoted by row and an attribute is denoted
>by a column which can dynamically change.)
>
>
>-
>
>Node - Cluster configuration (all nodes are identical)
>
>
>1.
>
>High availability (No single point of failure. In case of node failure
>other identical nodes are available.)
>2.
>
>No data loss due to replication (identical nodes)
>3.
>
>Easy to model (column store is very similar to RDBMS tables.)
>4.
>
>Similar query language to SQL (CQL)
>5.
>
>High performance (No master node concept. Any node can provide
>requested query. No performance bottleneck.)
>6.
>
>Transaction implementation
>
>
>1.
>
>Does not support queue structure (message queue implementation is an
>anti-pattern : http://www.datastax.com/dev/
>blog/cassandra-anti-patterns-queues-and-queue-like-datasets
>
> 
>)
>
> Redis
>
>
>-
>
>A simple key value storage
>-
>
>In memory database
>-
>
>Support data structures like lists, caped lists, hashes, sets etc. (
>https://redis.io/topics/data-types-intro
>)
>
>
>1.
>
>Fast read,write performances
>2.
>
>Provide persistence by writing to hard disk in configurable intervals.
>(snapshots : https://redis.io/topics/persistence)
>3.
>
>Can implement message queue structures. (http://fiznool.com/blog/2016/
>02/24/building-a-simple-message-queue-with-redis/
>
> 
>)
>4.
>
>Use to implement message stores. (https://redis.io/topics/pubsub)
>5.
>
>In built basic transaction support (MULTI/EXEC commands :
>https://redis.io/topics/transactions
>)
>
>
>1.
>
>Loss of data (in case of power outage etc.)
>2.
>
>Depends on CPU performances when data amount is too high (
>https://redis.io/topics/persistence
>)
>
> RocksDB
>
>
>-
>
>A simple key value storage (upgrade of LevelDB)
>-
>
>In memory database
>
>
>1.
>
>Fast read,write performances (faster than LevelDB)
>2.
>
>Can implement queue structures (https://github.com/facebook/
>rocksdb/wiki/Implement-Queue-Service-Using-RocksDB
>
> 

Re: [Architecture] File Based Database for Message Broker Store

[Indika Sampath]

Hi Wishmitha,

No SQL database is not an option as we have prior experience with
Cassandra. Basically, the important part is to find out performance and
fault tolerance capability in the database. Message broker
read/write/delete records all the time and database should be able to
handle high concurrency. Some metadata and message content write as byte
data. Also, message order matters when delivery. Hence database should
capable to sort records rather than bringing that logic into broker code.
As you mentioned transaction is another factor we are looking in the
database. Shall we start POC with Redis and see how it goes?

Cheers!

On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis 
wrote:

> Hi all,
>
> I am working on a project to replace the current RDBMS based database of
> the message broker store with a file based database system. I have been
> researching on potential file based databases which can be used in the
> project scenario. I have evaluated the pros and cons of the each of the
> potential databases mainly based on read and write performances,
> transaction support, data structure (queue) implementation support and
> overhead of replacement.
>
> A summary of database evaluation is represented in the following table.
>
>
> Database
>
> Pros
>
> Cons
>
> LevelDB
>
>-
>
>A simple key value storage
>-
>
>Used in ActiveMQ (replaced with KahaDB)
>
>
>1.
>
>Fast due to sorted keys
>2.
>
>Simple transaction support with batch( ) operation
>3.
>
>Sublevel feature (provides more organization and transactions
>facilitation)
>4.
>
>LiveStream feature (able to query things which are yet being receiving)
>5.
>
>Support triggers, groupby (mapreduce) and join (fanout) etc.
>
>
>1.
>
>Performance decrease in concurrent / multi threaded scenarios.
>(improved in RocksDB.)
>2.
>
>No object relational mapping.
>3.
>
>Poor performance when memory is not enough.
>
> MongoDB
>
>
>-
>
>Document oriented
>-
>
>Can define JSON schemas
>
>
>
>1.
>
>Direct object mapping (good replacement for RDBMS)
>2.
>
>Can implement queue like data structures (store messages in separate
>documents and delete them once delivered / already implemented data
>structures : https://www.npmjs.com/package/mongodb-queue)
>3.
>
>Capped collections (automatically remove old docs to make space for
>new docs.)
>
>
>1.
>
>No transaction support for multiple documents.
>2.
>
>Low performance compared to other NoSQL databases.
>
> Caassandra
>
>
>-
>
>Column oriented (a key is denoted by row and an attribute is denoted
>by a column which can dynamically change.)
>
>
>-
>
>Node - Cluster configuration (all nodes are identical)
>
>
>1.
>
>High availability (No single point of failure. In case of node failure
>other identical nodes are available.)
>2.
>
>No data loss due to replication (identical nodes)
>3.
>
>Easy to model (column store is very similar to RDBMS tables.)
>4.
>
>Similar query language to SQL (CQL)
>5.
>
>High performance (No master node concept. Any node can provide
>requested query. No performance bottleneck.)
>6.
>
>Transaction implementation
>
>
>1.
>
>Does not support queue structure (message queue implementation is an
>anti-pattern : http://www.datastax.com/dev/
>blog/cassandra-anti-patterns-queues-and-queue-like-datasets
>
> 
>)
>
> Redis
>
>
>-
>
>A simple key value storage
>-
>
>In memory database
>-
>
>Support data structures like lists, caped lists, hashes, sets etc. (
>https://redis.io/topics/data-types-intro
>)
>
>
>1.
>
>Fast read,write performances
>2.
>
>Provide persistence by writing to hard disk in configurable intervals.
>(snapshots : https://redis.io/topics/persistence)
>3.
>
>Can implement message queue structures. (http://fiznool.com/blog/2016/
>02/24/building-a-simple-message-queue-with-redis/
>
> 
>)
>4.
>
>Use to implement message stores. (https://redis.io/topics/pubsub)
>5.
>
>In built basic transaction support (MULTI/EXEC commands :
>https://redis.io/topics/transactions
>)
>
>
>1.
>
>Loss of data (in case of power outage etc.)
>2.
>
>Depends on CPU performances when data amount is too high (
>https://redis.io/topics/persistence
>)
>
> RocksDB
>
>
>-
>
>A simple key value storage (upgrade of LevelDB)
>-
>
>In memory database
>
>
>1.
>
>Fast read,write performances (faster than LevelDB)
>2.
>
>Can implement queue structures (https:/

[Architecture] File Based Database for Message Broker Store

[Wishmitha Mendis]

Hi all,

I am working on a project to replace the current RDBMS based database of
the message broker store with a file based database system. I have been
researching on potential file based databases which can be used in the
project scenario. I have evaluated the pros and cons of the each of the
potential databases mainly based on read and write performances,
transaction support, data structure (queue) implementation support and
overhead of replacement.

A summary of database evaluation is represented in the following table.


Database

Pros

Cons

LevelDB

   -

   A simple key value storage
   -

   Used in ActiveMQ (replaced with KahaDB)


   1.

   Fast due to sorted keys
   2.

   Simple transaction support with batch( ) operation
   3.

   Sublevel feature (provides more organization and transactions
   facilitation)
   4.

   LiveStream feature (able to query things which are yet being receiving)
   5.

   Support triggers, groupby (mapreduce) and join (fanout) etc.


   1.

   Performance decrease in concurrent / multi threaded scenarios. (improved
   in RocksDB.)
   2.

   No object relational mapping.
   3.

   Poor performance when memory is not enough.

MongoDB


   -

   Document oriented
   -

   Can define JSON schemas



   1.

   Direct object mapping (good replacement for RDBMS)
   2.

   Can implement queue like data structures (store messages in separate
   documents and delete them once delivered / already implemented data
   structures : https://www.npmjs.com/package/mongodb-queue)
   3.

   Capped collections (automatically remove old docs to make space for new
   docs.)


   1.

   No transaction support for multiple documents.
   2.

   Low performance compared to other NoSQL databases.

Caassandra


   -

   Column oriented (a key is denoted by row and an attribute is denoted by
   a column which can dynamically change.)


   -

   Node - Cluster configuration (all nodes are identical)


   1.

   High availability (No single point of failure. In case of node failure
   other identical nodes are available.)
   2.

   No data loss due to replication (identical nodes)
   3.

   Easy to model (column store is very similar to RDBMS tables.)
   4.

   Similar query language to SQL (CQL)
   5.

   High performance (No master node concept. Any node can provide requested
   query. No performance bottleneck.)
   6.

   Transaction implementation


   1.

   Does not support queue structure (message queue implementation is an
   anti-pattern :
   
http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets
   )

Redis


   -

   A simple key value storage
   -

   In memory database
   -

   Support data structures like lists, caped lists, hashes, sets etc. (
   https://redis.io/topics/data-types-intro)


   1.

   Fast read,write performances
   2.

   Provide persistence by writing to hard disk in configurable intervals.
   (snapshots : https://redis.io/topics/persistence)
   3.

   Can implement message queue structures. (
   
http://fiznool.com/blog/2016/02/24/building-a-simple-message-queue-with-redis/
   )
   4.

   Use to implement message stores. (https://redis.io/topics/pubsub)
   5.

   In built basic transaction support (MULTI/EXEC commands :
   https://redis.io/topics/transactions)


   1.

   Loss of data (in case of power outage etc.)
   2.

   Depends on CPU performances when data amount is too high (
   https://redis.io/topics/persistence)

RocksDB


   -

   A simple key value storage (upgrade of LevelDB)
   -

   In memory database


   1.

   Fast read,write performances (faster than LevelDB)
   2.

   Can implement queue structures (
   
https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB
   )
   3.

   Support concurrency
   4.

   Highly configurable (Pluggable Architecture)
   5.

   Support persistence (memtables and transactional logs manage in memory.
   Memtable is flushed to a sstfile to provide persistence.)


   1.

   Loss of data (in case of power outage etc.)
   2.

   No in built transaction support (have to use TansactionDB :
   https://github.com/facebook/rocksdb/wiki/Transactions )


According to this evaluation I suggest Redis as the most suitable database
for the message broker store. Even though it has an element of risk in data
loss, the persistence storing is configurable unlike in other key-value
stores. And it provides fast read and write performances compared to other
databases with basic transaction support.

And this is basically my evaluation and the opinion. It would be great to
have a feedback on this, specially regarding the additional criteria to
consider and other suggested databases.

Best Regards

-- 

*Wishmitha Mendis*

*Intern - Software Engineering*
*WSO2*

*Mobile : +94 777577706*
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture