hi all,
>> *@Nalaka,* >> Have we done a load test to verify whether this feature does not affect >> to the functionalities when there is a high load in a high concurrency >> situation? >> If so appreciate if you can share that details as well. >> > > Yes, we did. @Nalaka Senarathna <nala...@wso2.com> please point to the > relevant thread. > Earlier we tested this feature to verify the growing Audit table is reduced performance or not( [1] ) This feature is further tested with followings 1.Generate refresh revoke 2.Generate revoke 3.Generate refresh 4.Generate Testing Phase 1: with feature(Storing the old access tokens in audit table ) Tested with 100 threads using 100 users Token generate: average throughput 200/s Generate and refresh: average throughput 170/s Generate refresh revoke: average throughput 188/s Tested with 10 threads Generate revoke: average throughput 840/s Tested until 2 million in the audit table, 20000 in access token table. Latency doesn’t increase when the audit table is large. Testing phase 2: without this feature(Storing the old access tokens in the same table) Perform well in starting and reduce the throughputs when the table is growing. Clearly observed in token generate then refresh then revoke: throughput 55/s. Latency increasing when access token table is large. [1]Access token Table cleaning and keeping the access token data for future purposes thanks. On Sun, Jul 29, 2018 at 5:11 AM, Nuwan Dias <nuw...@wso2.com> wrote: > > > On Tue, Jul 24, 2018 at 3:17 PM Ishara Cooray <isha...@wso2.com> wrote: > >> Our objective is to run this clean up process without anyone having to >> configure anything. Even today we have instructions and scripts to clean up >> the tables. But in practice, nobody even notices this and never does these >> kind of stuff until they hit an issue and consult us through support. So if >> we design this in such a way that someone has to turn on something or >> configure something, I am positive our efforts will be in waste. >> >> So by default are we going to enable the token cleanup feature and >> storing old access tokens in audit table? >> > > Yes, otherwise there's no point of this feature IMO because we currently > have a solution that cleans data but in reality nobody uses it unless they > hit a issue and request our assistance to find and run this. > >> >> *@Nalaka,* >> Have we done a load test to verify whether this feature does not affect >> to the functionalities when there is a high load in a high concurrency >> situation? >> If so appreciate if you can share that details as well. >> > > Yes, we did. @Nalaka Senarathna <nala...@wso2.com> please point to the > relevant thread. > >> >> >> >> Thanks & Regards, >> Ishara Cooray >> Senior Software Engineer >> Mobile : +9477 262 9512 >> WSO2, Inc. | http://wso2.com/ >> Lean . Enterprise . Middleware >> >> On Fri, Jul 20, 2018 at 10:04 AM, Harshan Liyanage <hars...@wso2.com> >> wrote: >> >>> Hi Nuwan, >>> >>> Thanks for your detailed clarifications. Both explanations are perfectly >>> valid. >>> >>> Regards, >>> >>> Harshan Liyanage >>> Mobile: *+94765672894* >>> Email: hars...@wso2.com >>> Blog: http://harshanliyanage.blogspot.com/ >>> Medium: https://medium.com/@harshan.dll >>> *WSO2, Inc.:** wso2.com <http://wso2.com/>* >>> lean.enterprise.middleware. >>> >>> >>> On Thu, Jul 19, 2018 at 8:35 PM Nuwan Dias <nuw...@wso2.com> wrote: >>> >>>> >>>> >>>> On Wed, Jul 18, 2018 at 9:09 PM Harshan Liyanage <hars...@wso2.com> >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> @Nuwan: That's why I suggested having a configurable cron expression >>>>> so that users can configure the task to run on an optimal schedule instead >>>>> of blocking vital functionalities. Also in that way, we could do a batch >>>>> deletes and insertions instead of single rows. For example, they could let >>>>> it run every mid-night so then the access token tables won't grow for >>>>> millions of records and not affecting the user traffic. >>>>> >>>> >>>> Our objective is to run this clean up process without anyone having to >>>> configure anything. Even today we have instructions and scripts to clean up >>>> the tables. But in practice, nobody even notices this and never does these >>>> kind of stuff until they hit an issue and consult us through support. So if >>>> we design this in such a way that someone has to turn on something or >>>> configure something, I am positive our efforts will be in waste. >>>> >>>>> >>>>> @Nalaka: You could let the task to run only on a manager node in a >>>>> distributed setup using a configuration. >>>>> >>>> >>>> There is no such thing as a "manager" node in our architecture. All >>>> nodes are equal. Plus, we cannot introduce such changes as well due to >>>> numerous complexities such as how to HA the manager node, too many varying >>>> configurations, etc. >>>> >>>>> >>>>> BTW that was just a suggestion. It doesn't mean I'm -1 on this >>>>> proposed design. :) >>>>> >>>>> Thanks, >>>>> >>>>> Harshan Liyanage >>>>> Mobile: *+94765672894* >>>>> Email: hars...@wso2.com >>>>> Blog: http://harshanliyanage.blogspot.com/ >>>>> Medium: https://medium.com/@harshan.dll >>>>> *WSO2, Inc.:** wso2.com <http://wso2.com/>* >>>>> lean.enterprise.middleware. >>>>> >>>>> >>>>> On Wed, Jul 18, 2018 at 8:51 PM Nalaka Senarathna <nala...@wso2.com> >>>>> wrote: >>>>> >>>>>> hi harshan, >>>>>> >>>>>> Also if there are multiple nodes then those nodes also may attempt to >>>>>> clean up the same access token at the same time. >>>>>> >>>>>> related mail thread:[1] >>>>>> [1]Access token Table cleaning and keeping the access token data for >>>>>> future purposes >>>>>> >>>>>> regards. >>>>>> >>>>>> >>>>>> On Wed, Jul 18, 2018 at 8:28 PM, Nuwan Dias <nuw...@wso2.com> wrote: >>>>>> >>>>>>> A periodic task won't work for this because when the system runs on >>>>>>> tables with millions of records, the task will lock the table for the >>>>>>> clean >>>>>>> up process. This will impact other queries being executed on the table >>>>>>> and >>>>>>> hence block user flows. >>>>>>> >>>>>>> On Wed, Jul 18, 2018 at 6:17 AM Harshan Liyanage <hars...@wso2.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Nalaka, >>>>>>>> >>>>>>>> You could do the same with a configurable periodic task instead of >>>>>>>> modifying existing token request flows. What you have to do is to >>>>>>>> implement >>>>>>>> the token cleanup feature as a periodic task which scans the token >>>>>>>> related >>>>>>>> tables and move EXPIRED, INACTIVE and REVOKED tokens to the audit >>>>>>>> table. There will be a configuration to configure the CRON expression >>>>>>>> for >>>>>>>> that task. >>>>>>>> >>>>>>>> WDYT? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Harshan Liyanage >>>>>>>> Mobile: *+94765672894* >>>>>>>> Email: hars...@wso2.com >>>>>>>> Blog: http://harshanliyanage.blogspot.com/ >>>>>>>> Medium: https://medium.com/@harshan.dll >>>>>>>> *WSO2, Inc.:** wso2.com <http://wso2.com/>* >>>>>>>> lean.enterprise.middleware. >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Jul 18, 2018 at 2:45 PM Nalaka Senarathna <nala...@wso2.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> A solution for the access token table filled up with EXPIRED, >>>>>>>>> INACTIVE and REVOKED tokens in the Access token table, old access >>>>>>>>> token can >>>>>>>>> move to the Audit table when the new token is generating, renewing or >>>>>>>>> token >>>>>>>>> revoking. >>>>>>>>> >>>>>>>>> >>>>>>>>> Old Access tokens will be stored in newly created Audit table for >>>>>>>>> audit purposes. >>>>>>>>> >>>>>>>>> >>>>>>>>> *Token cleanup feature can be turn enable / disable. >>>>>>>>> >>>>>>>>> *When the token cleanup feature is enabled old access token can be >>>>>>>>> stored in the audit table >>>>>>>>> >>>>>>>>> For audit purposes or can disable that feature too, not to >>>>>>>>> store old access tokens(this also can enable/disable). >>>>>>>>> >>>>>>>>> >>>>>>>>> This project avoids the access token table growing and improve the >>>>>>>>> token lookup. For existing users also can get advantage from this. >>>>>>>>> >>>>>>>>> >>>>>>>>> Growing Audit table is no problem if the token cleanup feature and >>>>>>>>> retain old access token feature is enabled .because writing operation >>>>>>>>> to >>>>>>>>> Audit table is not increased when the token table is growing(Audit >>>>>>>>> table >>>>>>>>> has no constraint).This property was tested until 14 million token >>>>>>>>> generation. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> regards. >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Nalaka Senarathna* >>>>>>>>> *Associate Software Engineer | WSO2* >>>>>>>>> >>>>>>>>> *Email : nala...@wso2.com <nala...@wso2.com>* >>>>>>>>> *Mobile : +94714118474* >>>>>>>>> *web : https://wso2.com <https://wso2.com>* >>>>>>>>> <https://wso2.com/signature> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Architecture mailing list >>>>>>>>> Architecture@wso2.org >>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Nuwan Dias >>>>>>> >>>>>>> Director - WSO2, Inc. http://wso2.com >>>>>>> email : nuw...@wso2.com >>>>>>> Phone : +94 777 775 729 >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Architecture mailing list >>>>>>> Architecture@wso2.org >>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Nalaka Senarathna* >>>>>> *Associate Software Engineer | WSO2* >>>>>> >>>>>> *Email : nala...@wso2.com <nala...@wso2.com>* >>>>>> *Mobile : +94714118474* >>>>>> *web : https://wso2.com <https://wso2.com>* >>>>>> <https://wso2.com/signature> >>>>>> >>>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> Architecture@wso2.org >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>> >>>> >>>> -- >>>> Nuwan Dias >>>> >>>> Director - WSO2, Inc. http://wso2.com >>>> email : nuw...@wso2.com >>>> Phone : +94 777 775 729 >>>> _______________________________________________ >>>> Architecture mailing list >>>> Architecture@wso2.org >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>> >>> _______________________________________________ >>> Architecture mailing list >>> Architecture@wso2.org >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> > > -- > Nuwan Dias > > Director - WSO2, Inc. http://wso2.com > email : nuw...@wso2.com > Phone : +94 777 775 729 > -- *Nalaka Senarathna* *Associate Software Engineer | WSO2* *Email : nala...@wso2.com <nala...@wso2.com>* *Mobile : +94714118474* *web : https://wso2.com <https://wso2.com>* <https://wso2.com/signature>
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
