I'm Jose Maria Alvarez, and I'm part of one of the biggest retailers in Spain, called "El Corte Ingles". As you may know, we have a project to include WSO API-GW in our current architecture, and as part of that, we faced a problem related to the fact that you don't support MongoDB as the database backend for analytics.
As this NOSQL database is very popular in our company, and has a wide support in-house and outside, we have been working in developing a datasource for MongoDB to store analytics events inside that. We think we have finished the basic implementation, as all the tests are OK now (apart from analysing how to integrate the different configuration options that Mongo has), and we would like to share the idea behind the implementation with you.
Basically, the principles are:
- All that is a table in the RDBMS world is going to be a collection in Mongo.
- There will be no support for partitioning inside the AnalyticsRecordStore, but left to sharding support in Mongo. We need to analyze the best key or compound key in the data to have better performance when writing, but this should be done apart from the code.
- We use the tenantId and the tableName to create the collection name, as you do in Cassandra implementation.
- The event will be saved with the default id created by mongo, and a field called arsId for the record ID field. The map with properties will be saved the same way it's saved in cassandra, leaving values as binaries.
The rest is pretty straightforward. We have tested it also in a real installation (two docker instances, one for the API-GW, one for the APIM-Analytics), and we have done pull requests in carbon-data and carbon-analytics repositories to start analyzing this:
Could you share with me your thoughts about the principles? Are they OK from your perspective?
As you may know, Mongo has a 16MB limit for a document (record). Do you think that a real event in WSO2 can exceed that size? I've been speaking with Isabelle, that is helping us integrate the product, and she told us the events are very small (apart from any custom event we would like to store), but I would also like to get your thoughts on that.
Thank you,
Jose Maria.
www.elcorteingles.es
-------------------------------------------------------------------------------------------------------------------
Este mensaje, y en su caso, cualquier fichero anexo al mismo, puede contener información confidencial, siendo para uso exclusivo del destinatario, quedando prohibida su divulgación copia o distribución a terceros sin la autorización expresa del remitente. Si Vd. ha recibido este mensaje erróneamente, se ruega lo notifique al remitente y proceda a su borrado. Gracias por su colaboración.
This message (including any attachments) may contain confidential information. It is intended for use by the recipient only. Any dissemination, copying or distribution to third parties without the express consent of the sender is strictly prohibited. If you have received this message in error, please delete it immediately and notify the sender. Thank you for your collaboration.
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
