Re: [Architecture] Old access token cleanup when token generating, renewing and revoking.

[Nalaka Senarathna]

hi all,



>> *@Nalaka,*
>> Have we done a load test to verify whether this feature does not affect
>> to the functionalities when there is a high load in a high concurrency
>> situation?
>> If so appreciate if you can share that details as well.
>>
>
> Yes, we did. @Nalaka Senarathna  please point to the
> relevant thread.
>


Earlier we tested this feature to verify the growing Audit table is reduced
performance or not( [1] )



This feature is further tested with followings


1.Generate refresh revoke

2.Generate revoke

3.Generate refresh

4.Generate



Testing Phase 1: with feature(Storing the old access tokens in audit table )


Tested with 100 threads using 100 users

Token generate: average throughput 200/s


Generate and refresh: average throughput 170/s


Generate refresh revoke: average throughput 188/s


Tested with 10 threads

Generate revoke: average throughput 840/s


Tested until 2 million in the audit table, 2 in access token table.

Latency doesn’t increase when the audit table is large.



Testing phase 2: without this feature(Storing the old access tokens in the
same table)


Perform well in starting and reduce the throughputs when the table is
growing.

Clearly observed in token generate then refresh then revoke: throughput
55/s.

Latency increasing when access token table is large.



[1]Access token Table cleaning and keeping the access token data for future
purposes



thanks.


On Sun, Jul 29, 2018 at 5:11 AM, Nuwan Dias  wrote:

>
>
> On Tue, Jul 24, 2018 at 3:17 PM Ishara Cooray  wrote:
>
>> Our objective is to run this clean up process without anyone having to
>> configure anything. Even today we have instructions and scripts to clean up
>> the tables. But in practice, nobody even notices this and never does these
>> kind of stuff until they hit an issue and consult us through support. So if
>> we design this in such a way that someone has to turn on something or
>> configure something, I am positive our efforts will be in waste.
>>
>> So by default are we going to enable the token cleanup feature and
>> storing old access tokens in audit table?
>>
>
> Yes, otherwise there's no point of this feature IMO because we currently
> have a solution that cleans data but in reality nobody uses it unless they
> hit a issue and request our assistance to find and run this.
>
>>
>> *@Nalaka,*
>> Have we done a load test to verify whether this feature does not affect
>> to the functionalities when there is a high load in a high concurrency
>> situation?
>> If so appreciate if you can share that details as well.
>>
>
> Yes, we did. @Nalaka Senarathna  please point to the
> relevant thread.
>
>>
>>
>>
>> Thanks & Regards,
>> Ishara Cooray
>> Senior Software Engineer
>> Mobile : +9477 262 9512
>> WSO2, Inc. | http://wso2.com/
>> Lean . Enterprise . Middleware
>>
>> On Fri, Jul 20, 2018 at 10:04 AM, Harshan Liyanage 
>> wrote:
>>
>>> Hi Nuwan,
>>>
>>> Thanks for your detailed clarifications. Both explanations are perfectly
>>> valid.
>>>
>>> Regards,
>>>
>>> Harshan Liyanage
>>> Mobile: *+94765672894*
>>> Email: hars...@wso2.com
>>> Blog: http://harshanliyanage.blogspot.com/
>>> Medium: https://medium.com/@harshan.dll
>>> *WSO2, Inc.:** wso2.com *
>>> lean.enterprise.middleware.
>>>
>>>
>>> On Thu, Jul 19, 2018 at 8:35 PM Nuwan Dias  wrote:
>>>


 On Wed, Jul 18, 2018 at 9:09 PM Harshan Liyanage 
 wrote:

> Hi all,
>
> @Nuwan: That's why I suggested having a configurable cron expression
> so that users can configure the task to run on an optimal schedule instead
> of blocking vital functionalities. Also in that way, we could do a batch
> deletes and insertions instead of single rows. For example, they could let
> it run every mid-night so then the access token tables won't grow for
> millions of records and not affecting the user traffic.
>

 Our objective is to run this clean up process without anyone having to
 configure anything. Even today we have instructions and scripts to clean up
 the tables. But in practice, nobody even notices this and never does these
 kind of stuff until they hit an issue and consult us through support. So if
 we design this in such a way that someone has to turn on something or
 configure something, I am positive our efforts will be in waste.

>
> @Nalaka: You could let the task to run only on a manager node in a
> distributed setup using a configuration.
>

 There is no such thing as a "manager" node in our architecture. All
 nodes are equal. Plus, we cannot introduce such changes as well due to
 numerous complexities such as how to HA the manager node, too many varying
 configurations, etc.

>
> BTW that was just a suggestion. It doesn't mean I'm -1 on this
> proposed design. :)
>
> Thanks,
>
> Harshan Liyanage
> Mobile: *+94765672894*
> Email: hars...@wso2.com
> Blog: 

Re: [Architecture] Old access token cleanup when token generating, renewing and revoking.

[Nuwan Dias]

On Tue, Jul 24, 2018 at 3:17 PM Ishara Cooray  wrote:

> Our objective is to run this clean up process without anyone having to
> configure anything. Even today we have instructions and scripts to clean up
> the tables. But in practice, nobody even notices this and never does these
> kind of stuff until they hit an issue and consult us through support. So if
> we design this in such a way that someone has to turn on something or
> configure something, I am positive our efforts will be in waste.
>
> So by default are we going to enable the token cleanup feature and storing
> old access tokens in audit table?
>

Yes, otherwise there's no point of this feature IMO because we currently
have a solution that cleans data but in reality nobody uses it unless they
hit a issue and request our assistance to find and run this.

>
> *@Nalaka,*
> Have we done a load test to verify whether this feature does not affect to
> the functionalities when there is a high load in a high concurrency
> situation?
> If so appreciate if you can share that details as well.
>

Yes, we did. @Nalaka Senarathna  please point to the
relevant thread.

>
>
>
> Thanks & Regards,
> Ishara Cooray
> Senior Software Engineer
> Mobile : +9477 262 9512
> WSO2, Inc. | http://wso2.com/
> Lean . Enterprise . Middleware
>
> On Fri, Jul 20, 2018 at 10:04 AM, Harshan Liyanage 
> wrote:
>
>> Hi Nuwan,
>>
>> Thanks for your detailed clarifications. Both explanations are perfectly
>> valid.
>>
>> Regards,
>>
>> Harshan Liyanage
>> Mobile: *+94765672894*
>> Email: hars...@wso2.com
>> Blog: http://harshanliyanage.blogspot.com/
>> Medium: https://medium.com/@harshan.dll
>> *WSO2, Inc.:** wso2.com *
>> lean.enterprise.middleware.
>>
>>
>> On Thu, Jul 19, 2018 at 8:35 PM Nuwan Dias  wrote:
>>
>>>
>>>
>>> On Wed, Jul 18, 2018 at 9:09 PM Harshan Liyanage 
>>> wrote:
>>>
 Hi all,

 @Nuwan: That's why I suggested having a configurable cron expression so
 that users can configure the task to run on an optimal schedule instead of
 blocking vital functionalities. Also in that way, we could do a batch
 deletes and insertions instead of single rows. For example, they could let
 it run every mid-night so then the access token tables won't grow for
 millions of records and not affecting the user traffic.

>>>
>>> Our objective is to run this clean up process without anyone having to
>>> configure anything. Even today we have instructions and scripts to clean up
>>> the tables. But in practice, nobody even notices this and never does these
>>> kind of stuff until they hit an issue and consult us through support. So if
>>> we design this in such a way that someone has to turn on something or
>>> configure something, I am positive our efforts will be in waste.
>>>

 @Nalaka: You could let the task to run only on a manager node in a
 distributed setup using a configuration.

>>>
>>> There is no such thing as a "manager" node in our architecture. All
>>> nodes are equal. Plus, we cannot introduce such changes as well due to
>>> numerous complexities such as how to HA the manager node, too many varying
>>> configurations, etc.
>>>

 BTW that was just a suggestion. It doesn't mean I'm -1 on this proposed
 design. :)

 Thanks,

 Harshan Liyanage
 Mobile: *+94765672894*
 Email: hars...@wso2.com
 Blog: http://harshanliyanage.blogspot.com/
 Medium: https://medium.com/@harshan.dll
 *WSO2, Inc.:** wso2.com *
 lean.enterprise.middleware.


 On Wed, Jul 18, 2018 at 8:51 PM Nalaka Senarathna 
 wrote:

> hi harshan,
>
> Also if there are multiple nodes then those nodes also may attempt to
> clean up the same access token at the same time.
>
> related mail thread:[1]
> [1]Access token Table cleaning and keeping the access token data for
> future purposes
>
> regards.
>
>
> On Wed, Jul 18, 2018 at 8:28 PM, Nuwan Dias  wrote:
>
>> A periodic task won't work for this because when the system runs on
>> tables with millions of records, the task will lock the table for the 
>> clean
>> up process. This will impact other queries being executed on the table 
>> and
>> hence block user flows.
>>
>> On Wed, Jul 18, 2018 at 6:17 AM Harshan Liyanage 
>> wrote:
>>
>>> Hi Nalaka,
>>>
>>> You could do the same with a configurable periodic task instead of
>>> modifying existing token request flows. What you have to do is to 
>>> implement
>>> the token cleanup feature as a periodic task which scans the token 
>>> related
>>> tables and move EXPIRED, INACTIVE and REVOKED tokens to the audit
>>> table. There will be a configuration to configure the CRON expression 
>>> for
>>> that task.
>>>
>>> WDYT?
>>>
>>> Thanks,
>>>
>>> Harshan Liyanage
>>> Mobile: *+94765672894*
>>> Email: 

Re: [Architecture] Old access token cleanup when token generating, renewing and revoking.

[Ishara Cooray]

Our objective is to run this clean up process without anyone having to
configure anything. Even today we have instructions and scripts to clean up
the tables. But in practice, nobody even notices this and never does these
kind of stuff until they hit an issue and consult us through support. So if
we design this in such a way that someone has to turn on something or
configure something, I am positive our efforts will be in waste.

So by default are we going to enable the token cleanup feature and storing
old access tokens in audit table?

*@Nalaka,*
Have we done a load test to verify whether this feature does not affect to
the functionalities when there is a high load in a high concurrency
situation?
If so appreciate if you can share that details as well.



Thanks & Regards,
Ishara Cooray
Senior Software Engineer
Mobile : +9477 262 9512
WSO2, Inc. | http://wso2.com/
Lean . Enterprise . Middleware

On Fri, Jul 20, 2018 at 10:04 AM, Harshan Liyanage  wrote:

> Hi Nuwan,
>
> Thanks for your detailed clarifications. Both explanations are perfectly
> valid.
>
> Regards,
>
> Harshan Liyanage
> Mobile: *+94765672894*
> Email: hars...@wso2.com
> Blog: http://harshanliyanage.blogspot.com/
> Medium: https://medium.com/@harshan.dll
> *WSO2, Inc.:** wso2.com *
> lean.enterprise.middleware.
>
>
> On Thu, Jul 19, 2018 at 8:35 PM Nuwan Dias  wrote:
>
>>
>>
>> On Wed, Jul 18, 2018 at 9:09 PM Harshan Liyanage 
>> wrote:
>>
>>> Hi all,
>>>
>>> @Nuwan: That's why I suggested having a configurable cron expression so
>>> that users can configure the task to run on an optimal schedule instead of
>>> blocking vital functionalities. Also in that way, we could do a batch
>>> deletes and insertions instead of single rows. For example, they could let
>>> it run every mid-night so then the access token tables won't grow for
>>> millions of records and not affecting the user traffic.
>>>
>>
>> Our objective is to run this clean up process without anyone having to
>> configure anything. Even today we have instructions and scripts to clean up
>> the tables. But in practice, nobody even notices this and never does these
>> kind of stuff until they hit an issue and consult us through support. So if
>> we design this in such a way that someone has to turn on something or
>> configure something, I am positive our efforts will be in waste.
>>
>>>
>>> @Nalaka: You could let the task to run only on a manager node in a
>>> distributed setup using a configuration.
>>>
>>
>> There is no such thing as a "manager" node in our architecture. All nodes
>> are equal. Plus, we cannot introduce such changes as well due to numerous
>> complexities such as how to HA the manager node, too many varying
>> configurations, etc.
>>
>>>
>>> BTW that was just a suggestion. It doesn't mean I'm -1 on this proposed
>>> design. :)
>>>
>>> Thanks,
>>>
>>> Harshan Liyanage
>>> Mobile: *+94765672894*
>>> Email: hars...@wso2.com
>>> Blog: http://harshanliyanage.blogspot.com/
>>> Medium: https://medium.com/@harshan.dll
>>> *WSO2, Inc.:** wso2.com *
>>> lean.enterprise.middleware.
>>>
>>>
>>> On Wed, Jul 18, 2018 at 8:51 PM Nalaka Senarathna 
>>> wrote:
>>>
 hi harshan,

 Also if there are multiple nodes then those nodes also may attempt to
 clean up the same access token at the same time.

 related mail thread:[1]
 [1]Access token Table cleaning and keeping the access token data for
 future purposes

 regards.


 On Wed, Jul 18, 2018 at 8:28 PM, Nuwan Dias  wrote:

> A periodic task won't work for this because when the system runs on
> tables with millions of records, the task will lock the table for the 
> clean
> up process. This will impact other queries being executed on the table and
> hence block user flows.
>
> On Wed, Jul 18, 2018 at 6:17 AM Harshan Liyanage 
> wrote:
>
>> Hi Nalaka,
>>
>> You could do the same with a configurable periodic task instead of
>> modifying existing token request flows. What you have to do is to 
>> implement
>> the token cleanup feature as a periodic task which scans the token 
>> related
>> tables and move EXPIRED, INACTIVE and REVOKED tokens to the audit
>> table. There will be a configuration to configure the CRON expression for
>> that task.
>>
>> WDYT?
>>
>> Thanks,
>>
>> Harshan Liyanage
>> Mobile: *+94765672894*
>> Email: hars...@wso2.com
>> Blog: http://harshanliyanage.blogspot.com/
>> Medium: https://medium.com/@harshan.dll
>> *WSO2, Inc.:** wso2.com *
>> lean.enterprise.middleware.
>>
>>
>> On Wed, Jul 18, 2018 at 2:45 PM Nalaka Senarathna 
>> wrote:
>>
>>>
>>> A solution for the access token table filled up with EXPIRED,
>>> INACTIVE and REVOKED tokens in the Access token table, old access token 
>>> can
>>> move to the Audit table when the new token is 

Re: [Architecture] Old access token cleanup when token generating, renewing and revoking.

[Harshan Liyanage]

Hi Nuwan,

Thanks for your detailed clarifications. Both explanations are perfectly
valid.

Regards,

Harshan Liyanage
Mobile: *+94765672894*
Email: hars...@wso2.com
Blog: http://harshanliyanage.blogspot.com/
Medium: https://medium.com/@harshan.dll
*WSO2, Inc.:** wso2.com *
lean.enterprise.middleware.


On Thu, Jul 19, 2018 at 8:35 PM Nuwan Dias  wrote:

>
>
> On Wed, Jul 18, 2018 at 9:09 PM Harshan Liyanage  wrote:
>
>> Hi all,
>>
>> @Nuwan: That's why I suggested having a configurable cron expression so
>> that users can configure the task to run on an optimal schedule instead of
>> blocking vital functionalities. Also in that way, we could do a batch
>> deletes and insertions instead of single rows. For example, they could let
>> it run every mid-night so then the access token tables won't grow for
>> millions of records and not affecting the user traffic.
>>
>
> Our objective is to run this clean up process without anyone having to
> configure anything. Even today we have instructions and scripts to clean up
> the tables. But in practice, nobody even notices this and never does these
> kind of stuff until they hit an issue and consult us through support. So if
> we design this in such a way that someone has to turn on something or
> configure something, I am positive our efforts will be in waste.
>
>>
>> @Nalaka: You could let the task to run only on a manager node in a
>> distributed setup using a configuration.
>>
>
> There is no such thing as a "manager" node in our architecture. All nodes
> are equal. Plus, we cannot introduce such changes as well due to numerous
> complexities such as how to HA the manager node, too many varying
> configurations, etc.
>
>>
>> BTW that was just a suggestion. It doesn't mean I'm -1 on this proposed
>> design. :)
>>
>> Thanks,
>>
>> Harshan Liyanage
>> Mobile: *+94765672894*
>> Email: hars...@wso2.com
>> Blog: http://harshanliyanage.blogspot.com/
>> Medium: https://medium.com/@harshan.dll
>> *WSO2, Inc.:** wso2.com *
>> lean.enterprise.middleware.
>>
>>
>> On Wed, Jul 18, 2018 at 8:51 PM Nalaka Senarathna 
>> wrote:
>>
>>> hi harshan,
>>>
>>> Also if there are multiple nodes then those nodes also may attempt to
>>> clean up the same access token at the same time.
>>>
>>> related mail thread:[1]
>>> [1]Access token Table cleaning and keeping the access token data for
>>> future purposes
>>>
>>> regards.
>>>
>>>
>>> On Wed, Jul 18, 2018 at 8:28 PM, Nuwan Dias  wrote:
>>>
 A periodic task won't work for this because when the system runs on
 tables with millions of records, the task will lock the table for the clean
 up process. This will impact other queries being executed on the table and
 hence block user flows.

 On Wed, Jul 18, 2018 at 6:17 AM Harshan Liyanage 
 wrote:

> Hi Nalaka,
>
> You could do the same with a configurable periodic task instead of
> modifying existing token request flows. What you have to do is to 
> implement
> the token cleanup feature as a periodic task which scans the token related
> tables and move EXPIRED, INACTIVE and REVOKED tokens to the audit
> table. There will be a configuration to configure the CRON expression for
> that task.
>
> WDYT?
>
> Thanks,
>
> Harshan Liyanage
> Mobile: *+94765672894*
> Email: hars...@wso2.com
> Blog: http://harshanliyanage.blogspot.com/
> Medium: https://medium.com/@harshan.dll
> *WSO2, Inc.:** wso2.com *
> lean.enterprise.middleware.
>
>
> On Wed, Jul 18, 2018 at 2:45 PM Nalaka Senarathna 
> wrote:
>
>>
>> A solution for the access token table filled up with EXPIRED,
>> INACTIVE and REVOKED tokens in the Access token table, old access token 
>> can
>> move to the Audit table when the new token is generating, renewing or 
>> token
>> revoking.
>>
>>
>> Old Access tokens will be stored in newly created Audit table for
>> audit purposes.
>>
>>
>> *Token cleanup feature can be turn enable / disable.
>>
>> *When the token cleanup feature is enabled old access token can be
>> stored in the audit table
>>
>>   For audit purposes or can disable that feature too, not to store
>> old access tokens(this also can enable/disable).
>>
>>
>> This project avoids the access token table growing and improve the
>> token lookup. For existing users also can get advantage from this.
>>
>>
>> Growing Audit table is no problem if the token cleanup feature and
>> retain old access token feature is enabled .because writing operation to
>> Audit table is not increased when the token table is growing(Audit table
>> has no constraint).This property was tested until 14 million token
>> generation.
>>
>>
>>
>> ​
>>
>>
>>
>> regards.
>>
>> --
>> *Nalaka Senarathna*
>> *Associate 

Re: [Architecture] Old access token cleanup when token generating, renewing and revoking.

[Nuwan Dias]

On Wed, Jul 18, 2018 at 9:09 PM Harshan Liyanage  wrote:

> Hi all,
>
> @Nuwan: That's why I suggested having a configurable cron expression so
> that users can configure the task to run on an optimal schedule instead of
> blocking vital functionalities. Also in that way, we could do a batch
> deletes and insertions instead of single rows. For example, they could let
> it run every mid-night so then the access token tables won't grow for
> millions of records and not affecting the user traffic.
>

Our objective is to run this clean up process without anyone having to
configure anything. Even today we have instructions and scripts to clean up
the tables. But in practice, nobody even notices this and never does these
kind of stuff until they hit an issue and consult us through support. So if
we design this in such a way that someone has to turn on something or
configure something, I am positive our efforts will be in waste.

>
> @Nalaka: You could let the task to run only on a manager node in a
> distributed setup using a configuration.
>

There is no such thing as a "manager" node in our architecture. All nodes
are equal. Plus, we cannot introduce such changes as well due to numerous
complexities such as how to HA the manager node, too many varying
configurations, etc.

>
> BTW that was just a suggestion. It doesn't mean I'm -1 on this proposed
> design. :)
>
> Thanks,
>
> Harshan Liyanage
> Mobile: *+94765672894*
> Email: hars...@wso2.com
> Blog: http://harshanliyanage.blogspot.com/
> Medium: https://medium.com/@harshan.dll
> *WSO2, Inc.:** wso2.com *
> lean.enterprise.middleware.
>
>
> On Wed, Jul 18, 2018 at 8:51 PM Nalaka Senarathna 
> wrote:
>
>> hi harshan,
>>
>> Also if there are multiple nodes then those nodes also may attempt to
>> clean up the same access token at the same time.
>>
>> related mail thread:[1]
>> [1]Access token Table cleaning and keeping the access token data for
>> future purposes
>>
>> regards.
>>
>>
>> On Wed, Jul 18, 2018 at 8:28 PM, Nuwan Dias  wrote:
>>
>>> A periodic task won't work for this because when the system runs on
>>> tables with millions of records, the task will lock the table for the clean
>>> up process. This will impact other queries being executed on the table and
>>> hence block user flows.
>>>
>>> On Wed, Jul 18, 2018 at 6:17 AM Harshan Liyanage 
>>> wrote:
>>>
 Hi Nalaka,

 You could do the same with a configurable periodic task instead of
 modifying existing token request flows. What you have to do is to implement
 the token cleanup feature as a periodic task which scans the token related
 tables and move EXPIRED, INACTIVE and REVOKED tokens to the audit
 table. There will be a configuration to configure the CRON expression for
 that task.

 WDYT?

 Thanks,

 Harshan Liyanage
 Mobile: *+94765672894*
 Email: hars...@wso2.com
 Blog: http://harshanliyanage.blogspot.com/
 Medium: https://medium.com/@harshan.dll
 *WSO2, Inc.:** wso2.com *
 lean.enterprise.middleware.


 On Wed, Jul 18, 2018 at 2:45 PM Nalaka Senarathna 
 wrote:

>
> A solution for the access token table filled up with EXPIRED, INACTIVE
> and REVOKED tokens in the Access token table, old access token can move to
> the Audit table when the new token is generating, renewing or token
> revoking.
>
>
> Old Access tokens will be stored in newly created Audit table for
> audit purposes.
>
>
> *Token cleanup feature can be turn enable / disable.
>
> *When the token cleanup feature is enabled old access token can be
> stored in the audit table
>
>   For audit purposes or can disable that feature too, not to store
> old access tokens(this also can enable/disable).
>
>
> This project avoids the access token table growing and improve the
> token lookup. For existing users also can get advantage from this.
>
>
> Growing Audit table is no problem if the token cleanup feature and
> retain old access token feature is enabled .because writing operation to
> Audit table is not increased when the token table is growing(Audit table
> has no constraint).This property was tested until 14 million token
> generation.
>
>
>
> ​
>
>
>
> regards.
>
> --
> *Nalaka Senarathna*
> *Associate Software Engineer | WSO2*
>
> *Email : nala...@wso2.com *
> *Mobile : +94714118474*
> *web :  https://wso2.com *
> 
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>

>>>
>>> --
>>> Nuwan Dias
>>>
>>> Director - WSO2, Inc. http://wso2.com
>>> email : nuw...@wso2.com
>>> Phone : +94 777 775 729
>>>
>>>