Hi Dinusha,
How do we handle the cleanup tasks related to this feature? (Deleting .dat
files, removing entries from AM_USAGE_UPLOADED_FILES) . These tables will
grow with time (same as the auth token table issue currently we have in
APIM/IS). Are we handling it through code (remove entries/files after
reading or similar method) or running a separate script to remove them?.
Chamila.
On Wed, Jun 20, 2018 at 11:29 AM, Dinusha Dissanayake <dinus...@wso2.com>
wrote:
> Hi Sinthuja,
>
> What we are trying to do here is, persist the events coming from
> micro-gateway so that events would not be lost when publishing them to
> analytics.
> Microgateway should be able to run on its own. If analytics server is not
> present at the moment we are invoking APIs in microgateway, there should be
> a way to persist those data. Hence we are writing them into a file so that
> we can publish them into analytics later.
>
> This is similar approach what we have done in APIM 2.2.0 and in cloud with
> on premises gateway concept.
>
> On Fri, Jun 8, 2018 at 5:34 PM, Sinthuja Rajendran <sinth...@wso2.com>
> wrote:
>
>> Hi Dinusha,
>>
>> Can you explain what is the problem we are trying to solve with this
>> approach? I'm not clear based on our problem statement or use case based on
>> the explanation given above.
>>
>> Thanks,
>> Sinthuja.
>>
>> On Tue, Jun 5, 2018 at 10:35 AM Dinusha Dissanayake <dinus...@wso2.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> We've been working on analytics filter for APIM 2.5 microgateway.
>>> Following is the design we came up with.
>>>
>>> *1. Filtering the event data and writing them to a file.*
>>>
>>>
>>> From the request stream, the relevant fields will be extracted and
>>> requestDTO will be populated using those attributes.
>>> EventDTO consist of streamID, timestamp, metadata, correlation data and
>>> payload data.
>>> EventDTO can be populated using requestDTO.
>>>
>>> Likewise when the response filters are available, we can extract the
>>> necessary attributes and fill ExecutionTimeDTO and ResponseDTO. Then the
>>> eventDTO can be populated corresponding to response related attributes.
>>>
>>> Once EventDTO is populated, we publish those events in to event stream.
>>> A method to write those events to a file has been subscribed to the
>>> event stream at the gateway initialization process.
>>> Hence whenever a events stream gets an event, it will write that event
>>> to the file we are defining.
>>>
>>>
>>> *2. Event publishing from files to analytics server.*
>>>
>>> Files will be written with .dat extension.
>>> For example say "api-usage-data.dat".
>>>
>>> In APIM v2.2 micro gateway, this file would rotate if the file size
>>> exceeds 12mb or after a specific time interval. These values could be set
>>> as configurable values.
>>> Rolled file will be compressed with the timestamp.
>>> eg: *api-usage-data.{timstamp}.zip*
>>> This would reduce 12mb file into (approximately) 4kb file.
>>>
>>> Then this zip(s) will be uploaded and persisted in a DB. This will be
>>> done using ballerina tasks(to make it happen periodically).
>>> Then the relevant entries (files) will be read from the DB and populate
>>> the events by reading line by line of that files. Then those events will be
>>> published to the analytics server using existing data publisher.
>>>
>>>
>>>
>>>
>>> Thanks,
>>> DinushaD
>>>
>>>
>>>
>>>
>>> --
>>> Dinusha Dissanayake
>>> Software Engineer
>>> WSO2 Inc
>>> Mobile: +94712939439
>>> <https://wso2.com/signature>
>>> _______________________________________________
>>> Architecture mailing list
>>> Architecture@wso2.org
>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>>
>>
>>
>> --
>> *Sinthuja Rajendran*
>> Senior Technical Lead
>> WSO2, Inc.:http://wso2.com
>>
>> Blog: http://sinthu-rajan.blogspot.com/
>> Mobile: +94774273955
>>
>>
>>
>> _______________________________________________
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
> Dinusha Dissanayake
> Software Engineer
> WSO2 Inc
> Mobile: +94712939439
> <https://wso2.com/signature>
>
> _______________________________________________
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>
--
Regards,
Chamila Adhikarinayake
WSO2, Inc.
Mobile - +94712346437
Email - chami...@wso2.com
Blog - http://helpfromadhi.blogspot.com/
_______________________________________________
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
