Hi Chamila We are handling it through the code.
.dat will be compressed according to a configured time interval (say per every 10 mins). Then the zip will be uploaded to endpoint as mentioned above. If it is a success, then the zip will be deleted. Table clean up task will be running and cleaning the DB periodically. @Fazlan is working on that I think. On Thu, Jun 21, 2018 at 2:31 PM, Chamila Adhikarinayake <chami...@wso2.com> wrote: > Hi Dinusha, > > How do we handle the cleanup tasks related to this feature? (Deleting .dat > files, removing entries from AM_USAGE_UPLOADED_FILES) . These tables will > grow with time (same as the auth token table issue currently we have in > APIM/IS). Are we handling it through code (remove entries/files after > reading or similar method) or running a separate script to remove them?. > > > Chamila. > > > On Wed, Jun 20, 2018 at 11:29 AM, Dinusha Dissanayake <dinus...@wso2.com> > wrote: > >> Hi Sinthuja, >> >> What we are trying to do here is, persist the events coming from >> micro-gateway so that events would not be lost when publishing them to >> analytics. >> Microgateway should be able to run on its own. If analytics server is not >> present at the moment we are invoking APIs in microgateway, there should be >> a way to persist those data. Hence we are writing them into a file so that >> we can publish them into analytics later. >> >> This is similar approach what we have done in APIM 2.2.0 and in cloud >> with on premises gateway concept. >> >> On Fri, Jun 8, 2018 at 5:34 PM, Sinthuja Rajendran <sinth...@wso2.com> >> wrote: >> >>> Hi Dinusha, >>> >>> Can you explain what is the problem we are trying to solve with this >>> approach? I'm not clear based on our problem statement or use case based on >>> the explanation given above. >>> >>> Thanks, >>> Sinthuja. >>> >>> On Tue, Jun 5, 2018 at 10:35 AM Dinusha Dissanayake <dinus...@wso2.com> >>> wrote: >>> >>>> Hi all, >>>> >>>> We've been working on analytics filter for APIM 2.5 microgateway. >>>> Following is the design we came up with. >>>> >>>> *1. Filtering the event data and writing them to a file.* >>>> >>>> >>>> From the request stream, the relevant fields will be extracted and >>>> requestDTO will be populated using those attributes. >>>> EventDTO consist of streamID, timestamp, metadata, correlation data and >>>> payload data. >>>> EventDTO can be populated using requestDTO. >>>> >>>> Likewise when the response filters are available, we can extract the >>>> necessary attributes and fill ExecutionTimeDTO and ResponseDTO. Then the >>>> eventDTO can be populated corresponding to response related attributes. >>>> >>>> Once EventDTO is populated, we publish those events in to event stream. >>>> A method to write those events to a file has been subscribed to the >>>> event stream at the gateway initialization process. >>>> Hence whenever a events stream gets an event, it will write that event >>>> to the file we are defining. >>>> >>>> >>>> *2. Event publishing from files to analytics server.* >>>> >>>> Files will be written with .dat extension. >>>> For example say "api-usage-data.dat". >>>> >>>> In APIM v2.2 micro gateway, this file would rotate if the file size >>>> exceeds 12mb or after a specific time interval. These values could be set >>>> as configurable values. >>>> Rolled file will be compressed with the timestamp. >>>> eg: *api-usage-data.{timstamp}.zip* >>>> This would reduce 12mb file into (approximately) 4kb file. >>>> >>>> Then this zip(s) will be uploaded and persisted in a DB. This will be >>>> done using ballerina tasks(to make it happen periodically). >>>> Then the relevant entries (files) will be read from the DB and populate >>>> the events by reading line by line of that files. Then those events will be >>>> published to the analytics server using existing data publisher. >>>> >>>> >>>> >>>> >>>> Thanks, >>>> DinushaD >>>> >>>> >>>> >>>> >>>> -- >>>> Dinusha Dissanayake >>>> Software Engineer >>>> WSO2 Inc >>>> Mobile: +94712939439 >>>> <https://wso2.com/signature> >>>> _______________________________________________ >>>> Architecture mailing list >>>> Architecture@wso2.org >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>> >>> >>> -- >>> *Sinthuja Rajendran* >>> Senior Technical Lead >>> WSO2, Inc.:http://wso2.com >>> >>> Blog: http://sinthu-rajan.blogspot.com/ >>> Mobile: +94774273955 >>> >>> >>> >>> _______________________________________________ >>> Architecture mailing list >>> Architecture@wso2.org >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> Dinusha Dissanayake >> Software Engineer >> WSO2 Inc >> Mobile: +94712939439 >> <https://wso2.com/signature> >> >> _______________________________________________ >> Architecture mailing list >> Architecture@wso2.org >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > Regards, > Chamila Adhikarinayake > WSO2, Inc. > Mobile - +94712346437 > Email - chami...@wso2.com > Blog - http://helpfromadhi.blogspot.com/ > -- Dinusha Dissanayake Software Engineer WSO2 Inc Mobile: +94712939439 <https://wso2.com/signature>
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture