Hi Malith, Thanks for your input. I have changed the jmeter scripts according to you instructions. I was using a oracle docker instance in my local machine. I have changed it to a remote oracle database. Now the latency is much less. I will share all the performance numbers once I have finished collecting them for oracle, mssql, mysql and postgres databases.
Thanks! Rajith On Tue, Jan 24, 2017 at 9:46 AM, Malith Jayasinghe <mali...@wso2.com> wrote: > @Rajith > > As per the offline discussion we had regarding the performance evaluation > (using JMETER) of the two methods: > - use 2 separate thread groups for evaluating the performance of 2 DB > based search methods > - run each thread group sequentially > - run the test for a longer period under different concurrency level and > record the latency and TPS > > With regard to the long latencies you are noticing, we need to figure out > if this is related to the database query/queries or something else. To do a > quick test: simply log the execution time of the query/queries. If the > execution times are high (or have spikes) then we can try to optimize the > DB query. Otherwise we have to do some further investigations. > > Thanks > > Malith > > > > > > > On Sat, Jan 14, 2017 at 10:43 PM, Nuwan Dias <nuw...@wso2.com> wrote: > >> File system based indexers bring new challenges in the container world. >> It also poses challenges in HA (multinode) and multi regional deployments. >> Therefore if we're selecting a solr indexing approach, the only practical >> solution is to go with a solr server based architecture. >> >> Even with a solr server assisted architecture, we still have complexities >> when dealing with multi regional deployments. Also since API artifacts >> reside in the database, if we're loading search results from an index, we >> have to sync the permissions of the artifacts in the index too. Plus this >> makes the deployment complex. >> >> Given the complexities that have to be addressed in an indexing solution, >> I'd prefer to opt with the DB based search at least to start off with. >> Since APIs and their permissions are both maintained in the DB, it would be >> much simple if the search also works on that. Unlike in C4 this database >> will only have data of 1 tenant. Hence we're not expecting the API count to >> be in the thousands. Therefore I think searching through the database >> should be feasible. >> >> Thanks, >> NuwanD. >> >> On Sat, 14 Jan 2017 at 8:27 am, Chandana Napagoda <chand...@wso2.com> >> wrote: >> >>> Hi Rajith, >>> >>> I believe indexing based approach is more suitable for any search >>> solution because its design is to support fast search results. If you use >>> database search, you have to use multiple DB indexes to improve search >>> performance, and that will reduce the performance of insert operations. >>> >>> I think, REG_LOG kind of history table is not necessary for APIM >>> product, since it is totally aware of the APIM artifacts(APIs, Docs, forum >>> posts) and you can directly connect to the DB[1] from the text search >>> engine and index the resources. As per the Solr documentation, it is >>> capable of importing only the new additions(delta) for indexing. >>> >>> [1]. https://cwiki.apache.org/confluence/display/solr/Uploading+ >>> Structured+Data+Store+Data+with+the+Data+Import+Handler >>> >>> Regards, >>> Chandana >>> >>> >>> On Fri, Jan 13, 2017 at 11:44 AM, Rajith Roshan <raji...@wso2.com> >>> wrote: >>> >>> Hi all, >>> >>> We are currently evaluating how to perform full text search at the >>> database level for C5 based API Manager. We will be evaluating this for >>> different types of databases to find their implementation complexities and >>> limitations. >>> Other option available for us to use indexing based approach (use Solr) >>> >>> *Database full text search * >>> *Pros* >>> >>> - Less complications when using container based approach >>> - Clustering will require only database syncing. >>> - No need to maintain and ship external search engine. >>> >>> *Cons* >>> >>> - Implementation may vary significantly based on the database type >>> - There can be limitation in full text search for particular >>> database types (For ex: mysql full text support only prefix search) >>> - Queries will differ based on database type >>> - Document search will not be available, because they are stored as >>> blobs >>> >>> >>> *Indexing based approach * >>> *Pros* >>> >>> - Document search >>> - Search will be efficient (No need to access database) >>> >>> *Cons* >>> >>> - Since indexing data is written to file system , when going for >>> container based approach we would require mechanisms to file system >>> mounting >>> - Syncing indexers in a cluster would require something similar to >>> existing C4 based registry architecture (use of REG_LOG table) >>> - Maintaining (for ex: Version updates) and shipping external search >>> engine. >>> >>> Your valuable input regrading this is highly appreciated. >>> >>> Thanks! >>> Rajith >>> >>> -- >>> Rajith Roshan >>> Software Engineer, WSO2 Inc. >>> >>> >>> Mobile: +94-72-642-8350 <%2B94-71-554-8430> >>> >>> >>> >>> >>> >>> >>> >>> -- >>> *Chandana Napagoda* >>> Associate Technical Lead >>> WSO2 Inc. - http://wso2.org >>> >>> *Email : chand...@wso2.com <chand...@wso2.com>**Mobile : +94718169299 >>> <+94%2071%20816%209299>* >>> >>> *Blog : http://cnapagoda.blogspot.com >>> <http://cnapagoda.blogspot.com> | http://chandana.napagoda.com >>> <http://chandana.napagoda.com>* >>> >>> *Linkedin : http://www.linkedin.com/in/chandananapagoda >>> <http://www.linkedin.com/in/chandananapagoda>* >>> >>> >>> >>> >> _______________________________________________ >> Architecture mailing list >> Architecture@wso2.org >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > Malith Jayasinghe > > WSO2, Inc. (http://wso2.com) > Email :mali...@wso2.com > Mobile :0770704040 > Blog :https://medium.com/@malith.jayasinghe > <https://medium.com/@malith.jayasinghe> > Lean . Enterprise . Middleware > -- Rajith Roshan Software Engineer, WSO2 Inc. Mobile: +94-72-642-8350 <%2B94-71-554-8430>
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
