Hi Jose Maria, Thank you very much for your contribution and for the detailed explanation on how you have mapped the MongoDB concepts to implement. Analytics RecordStores. We will go through the two pull requests you have sent us and will share our thoughts.
Thanks, Gimantha On Mon, Jan 9, 2017 at 1:49 PM JOSE MARIA ALVAREZ FERNANDEZ < josemaria.alvarezfernan...@elcorteingles.es> wrote: > Hi, > > I'm Jose Maria Alvarez, and I'm part of one of the biggest retailers in > Spain, called "El Corte Ingles". As you may know, we have a project to > include WSO API-GW in our current architecture, and as part of that, we > faced a problem related to the fact that you don't support MongoDB as the > database backend for analytics. > As this NOSQL database is very popular in our company, and has a wide > support in-house and outside, we have been working in developing a > datasource for MongoDB to store analytics events inside that. We think we > have finished the basic implementation, as all the tests are OK now (apart > from analysing how to integrate the different configuration options that > Mongo has), and we would like to share the idea behind the implementation > with you. > > Basically, the principles are: > > - All that is a table in the RDBMS world is going to be a collection in > Mongo. > - There will be no support for partitioning inside > the AnalyticsRecordStore, but left to sharding support in Mongo. We need to > analyze the best key or compound key in the data to have better performance > when writing, but this should be done apart from the code. > - We use the tenantId and the tableName to create the collection name, as > you do in Cassandra implementation. > - The event will be saved with the default id created by mongo, and a > field called arsId for the record ID field. The map with properties will be > saved the same way it's saved in cassandra, leaving values as binaries. > > The rest is pretty straightforward. We have tested it also in a real > installation (two docker instances, one for the API-GW, one for the > APIM-Analytics), and we have done pull requests in carbon-data and > carbon-analytics repositories to start analyzing this: > > https://github.com/wso2/carbon-data/pull/133 > https://github.com/wso2/carbon-analytics/pull/225 > > Could you share with me your thoughts about the principles? Are they OK > from your perspective? > As you may know, Mongo has a 16MB limit for a document (record). Do you > think that a real event in WSO2 can exceed that size? I've been speaking > with Isabelle, that is helping us integrate the product, and she told us > the events are very small (apart from any custom event we would like to > store), but I would also like to get your thoughts on that. > > Thank you, > > Jose Maria. > > > > > > > www.elcorteingles.es > > > > > > > > > ------------------------------------------------------------------------------------------------------------------- > > > Este mensaje, y en su caso, cualquier fichero anexo al mismo, puede > > contener información confidencial, siendo para uso exclusivo del > > destinatario, quedando prohibida su divulgación copia o distribución a > > terceros sin la autorización expresa del remitente. Si Vd. ha recibido > > este mensaje erróneamente, se ruega lo notifique al remitente y > > proceda a su borrado. > > Gracias por su colaboración. > > > > This message (including any attachments) may contain confidential > > information. It is intended for use by the recipient only. Any > > dissemination, copying or distribution to third parties without the > > express consent of the sender is strictly prohibited. If you have > > received this message in error, please delete it immediately and > > notify the sender. > > Thank you for your collaboration. > > > > > > _______________________________________________ > > Dev mailing list > > Dev@wso2.org > > http://wso2.org/cgi-bin/mailman/listinfo/dev > >
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
