Hi Lakshman, That's is a good observation. So I believe this event should be either first one or the last one. Can you please do a another small test? Shutdown the BAM server. Delete the Cassandra data folder completely (You can find that in repository/databases/cassandra) Truncate the APIM related RDBMS tables. Run your script with 100 requests.
Please try to identify the message that going to miss. I think we need to check data sending part in APIM side? Also add a debug log message to publishEvent method with static counter and print that value. So it will be easy to identify the issue. Regards, Gihan On Wed, Oct 22, 2014 at 6:37 PM, Lakshman Udayakantha <lakshm...@wso2.com> wrote: > Hi Gihan, > > I sent 10000 requests with the preconditions you have mentioned. only one > record is missed in cassendra database for every time I sent 10000 requests > via APIM. > > Thanks > > On Wed, Oct 22, 2014 at 9:28 AM, Lakshman Udayakantha <lakshm...@wso2.com> > wrote: > >> Hi Gihan, >> >> I will do that test and get back to you. >> >> Thanks >> >> On Wed, Oct 22, 2014 at 9:23 AM, Inosh Goonewardena <in...@wso2.com> >> wrote: >> >>> Hi, >>> >>> On Wednesday, October 22, 2014, Lakshman Udayakantha <lakshm...@wso2.com> >>> wrote: >>> >>>> Hi Inosh, >>>> >>>> data in cassandra database are summized to a relational database. >>>> records in cassandra database and this relational table is same. so I think >>>> there is no problem with cassandra explorer. >>>> >>> >>> Ok understood. I have misread your previous email. >>> >>>> >>>> Thanks >>>> >>>> On Wed, Oct 22, 2014 at 9:14 AM, Inosh Goonewardena <in...@wso2.com> >>>> wrote: >>>> >>>>> Hi Lakshman, >>>>> >>>>> You mean data is therw in cassandra and cassandra explorer doesn't >>>>> show these data? Did you compare by using cassandra CLI tool? >>>>> >>>>> On Wednesday, October 22, 2014, Lakshman Udayakantha < >>>>> lakshm...@wso2.com> wrote: >>>>> >>>>>> HI Gihan , >>>>>> >>>>>> They are lost randomly. >>>>>> >>>>>> Thanks >>>>>> >>>>>> On Wed, Oct 22, 2014 at 9:00 AM, Gihan Anuruddha <gi...@wso2.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Lakshman, >>>>>>> >>>>>>> How many records did you send to BAM and how many events are lost? >>>>>>> >>>>>>> Regards, >>>>>>> Gihan >>>>>>> >>>>>>> On Wed, Oct 22, 2014 at 8:18 AM, Lakshman Udayakantha < >>>>>>> lakshm...@wso2.com> wrote: >>>>>>> >>>>>>>> Hi Gihan, >>>>>>>> >>>>>>>> These data in cassandra database are summarised by a hive script >>>>>>>> and insert into a relational database. I checked that relational >>>>>>>> database >>>>>>>> table also. Number of records in cassandra database and relational >>>>>>>> database >>>>>>>> table are same. Then it should not be a problem with cassandra >>>>>>>> explorer. >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> On Wed, Oct 22, 2014 at 7:53 AM, Manoj Gunawardena <man...@wso2.com >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> This can be an issue for other products as well. ES, AS and EMM >>>>>>>>> also publishing data to BAM. In ES we are using load balancing data >>>>>>>>> publisher. I also notice if we call stop method, sometimes data not >>>>>>>>> get >>>>>>>>> publish to BAM. We changed this to initiate data publisher at the >>>>>>>>> first >>>>>>>>> event and call data publishers stop method in the bundle activator's >>>>>>>>> stop >>>>>>>>> method. >>>>>>>>> Need to test with load. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Oct 21, 2014 at 10:07 PM, Gihan Anuruddha <gi...@wso2.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Lakshman, >>>>>>>>>> >>>>>>>>>> It should be in Cassandra if publishEvent method executes. How >>>>>>>>>> do you validate the data loss in Cassandra side? BAM embedded >>>>>>>>>> Cassandra explorer is a not good tool for that. Can you please >>>>>>>>>> verify that with tool like CQL or cassandra-cli? >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Gihan >>>>>>>>>> >>>>>>>>>> On Tue, Oct 21, 2014 at 9:18 PM, Lakshman Udayakantha < >>>>>>>>>> lakshm...@wso2.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi All, >>>>>>>>>>> >>>>>>>>>>> I am working on https://wso2.org/jira/browse/APIMANAGER-2064. >>>>>>>>>>> >>>>>>>>>>> By looking at the Cassandra database on BAM, I could figure out >>>>>>>>>>> some requests publish to Cassandra database and some do not when I >>>>>>>>>>> invoke >>>>>>>>>>> the api. That is the reason for the missing requests in statistics >>>>>>>>>>> dashboard. >>>>>>>>>>> >>>>>>>>>>> I tried debugging the APIMgtUsageDataBridgeDataPublisher class >>>>>>>>>>> since it has publishEvent method to publish data. >>>>>>>>>>> APIMgtUsageDataBridgeDataPublisher class uses >>>>>>>>>>> LoadBalancingDataPublisher as >>>>>>>>>>> the dataPublisher in this case. >>>>>>>>>>> >>>>>>>>>>> I found that every time when I invoke the api via rest client, >>>>>>>>>>> this method will execute. but randomly some requests are not >>>>>>>>>>> publishing in >>>>>>>>>>> Cassandra database. I used the >>>>>>>>>>> latest org.wso2.carbon.databridge.agent.thrift jar. but the problem >>>>>>>>>>> is >>>>>>>>>>> still there. >>>>>>>>>>> >>>>>>>>>>> if requests come to publishEvent method, should they publish in >>>>>>>>>>> Cassandra database?and is there any known reason for data get not >>>>>>>>>>> published >>>>>>>>>>> in cassendra database. if not where I can find the data publisher to >>>>>>>>>>> cassendra in BAM from API Manager. >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> -- >>>>>>>>>>> Lakshman Udayakantha >>>>>>>>>>> WSO2 Inc. www.wso2.com >>>>>>>>>>> lean.enterprise.middleware >>>>>>>>>>> Mobile: *0711241005 <0711241005>* >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Dev mailing list >>>>>>>>>>> Dev@wso2.org >>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> W.G. Gihan Anuruddha >>>>>>>>>> Senior Software Engineer | WSO2, Inc. >>>>>>>>>> M: +94772272595 >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Dev mailing list >>>>>>>>>> Dev@wso2.org >>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Manoj Gunawardena >>>>>>>>> Tech Lead >>>>>>>>> WSO2, Inc.: http://wso2.com >>>>>>>>> lean.enterprise.middleware >>>>>>>>> Mobile : +94 77 2291643 >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Lakshman Udayakantha >>>>>>>> WSO2 Inc. www.wso2.com >>>>>>>> lean.enterprise.middleware >>>>>>>> Mobile: *0711241005 <0711241005>* >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> W.G. Gihan Anuruddha >>>>>>> Senior Software Engineer | WSO2, Inc. >>>>>>> M: +94772272595 >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Lakshman Udayakantha >>>>>> WSO2 Inc. www.wso2.com >>>>>> lean.enterprise.middleware >>>>>> Mobile: *0711241005 <0711241005>* >>>>>> >>>>>> >>>>> >>>>> -- >>>>> Regards, >>>>> >>>>> Inosh Goonewardena >>>>> Associate Technical Lead- WSO2 Inc. >>>>> Mobile: +94779966317 >>>>> >>>>> >>>> >>>> >>>> -- >>>> Lakshman Udayakantha >>>> WSO2 Inc. www.wso2.com >>>> lean.enterprise.middleware >>>> Mobile: *0711241005 <0711241005>* >>>> >>>> >>> >>> -- >>> Regards, >>> >>> Inosh Goonewardena >>> Associate Technical Lead- WSO2 Inc. >>> Mobile: +94779966317 >>> >>> >> >> >> -- >> Lakshman Udayakantha >> WSO2 Inc. www.wso2.com >> lean.enterprise.middleware >> Mobile: *0711241005 <0711241005>* >> >> > > > -- > Lakshman Udayakantha > WSO2 Inc. www.wso2.com > lean.enterprise.middleware > Mobile: *0711241005 <0711241005>* > > -- W.G. Gihan Anuruddha Senior Software Engineer | WSO2, Inc. M: +94772272595
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
