Re: [Architecture] Security Authentication Analytics

[Damith Wickramasinghe]

Hi Sajith,

Sorry I have missed this. No for the first phase we are not focusing on
this.

Regards,
Damith.

On Tue, Mar 29, 2016 at 10:06 AM, Sajith Ravindra  wrote:

> Hi Damith,
>
> Do we support these statistics per tenant? So that each tenant can view
> these statistics per tenant.
>
> Thanks
> *,Sajith Ravindra*
> Senior Software Engineer
> WSO2 Inc.; http://wso2.com
> lean.enterprise.middleware
>
> mobile: +94 77 2273550
> blog: http://sajithr.blogspot.com/
> 
>
> On Mon, Mar 28, 2016 at 4:08 AM, Damith Wickramasinghe 
> wrote:
>
>> Sure.We will schedule.
>>
>> Regards,
>> Damith.
>>
>> On Mon, Mar 28, 2016 at 4:02 PM, Srinath Perera  wrote:
>>
>>> As per our chat, lets schedule a review.
>>>
>>> --Srinath
>>>
>>> On Sun, Mar 27, 2016 at 11:19 PM, Damith Wickramasinghe <
>>> dami...@wso2.com> wrote:
>>>
 Hi All,

 When it comes to Security analytics story we have broken it down to
 mainly three sections(Can be more in the future). Namely Authentication
 Analytics , Authorization Analytics and Audit trail for identity
 artifacts.  As for the first phase we are focusing on Authentication
 Analytics and Audit trail for identity artifacts.

 Data Summarization

 For Authentication Analytics when a user authenticates we are
 publishing related data from IS side for a raw table which will get
 persisted In DAS. Then using scheduled spark scripts we are summarizing the
 data to yearly,monthly,daily,hourly,minutely and secondly tables. (We may
 drop the secondly table since it may contain lot of data and will be not
 efficient when carrying out aggregate operations. Also these scheduled
 spark scripts will run incrementally without summarizing previous data
 again.)

 As per the [1] we have following charts.We are only considering login
 success and failure scenarios. (As discussed with IS team we dropped logout
 success and failure scenarios for now.Since showing those statistics are
 not much important.)

- overall authentication success and failure count during a time
range - Area chart
- per user Authentication success count - horizontal bar chart
- per user Authentication failure count - horizontal bar chart
- per role Authentication success count - horizontal bar chart
- per role Authentication failure count - horizontal bar chart

  etc. As above there are charts for service provider,identity
 provider,region and for ip ranges as well.

 Above Area chart for overall authentication success and failure count
 can be further drilled down as user clicks on horizontal bar charts(per
 user,per role etc). To cater this we have a one table structure with
 columns (if we take monthly table) Per month -> Per user -> Per roles ->
 Per serviceProvider -> Per identityProvider -> Per region -> Per Ip ->
 authSuccessCount and authFailureCount. Ill call this TABLE1.

 Please note in above table we are keeping the roles as comma separated
 values. so when a role is clicked we can identify authSuccess and
 authFailure count using DAS score function[2]. We discussed on getting this
 comma separated roles per event from IS directly since it will make things
 easier when it comes to summarization logic.

 For all other horizontal bar charts except role table , we are
 following below table structure.

 (If we take monthly table for user) Per month -> Per user
 ->authSuccessCount and authFailureCount . Ill call this TABLE2.

 Its true that these information can also be achieved from TABLE1. But
 since it has multi level grouped data, data aggregation will take more
 time. So having on level grouping will allow less records to be aggregated.

 For the role table we need a row for each role with corresponding
 authSuccessCount and authFailureCount. But as mentioned above since we are
 sending roles as comma separated values we do not have a efficient way to
 separate each role and construct the table. So we thought of getting the
 data as duplicated events(per user will have multiple roles so a event will
 be duplicated because of the role) from IS side and do the summarization.

 (If we take monthly table per role) Per Role -> Per User -> Per Service
 provider -> Per identity Provider -> Per region -> Per IP ->
 authSuccessCount and authFailureCount. Ill call this TABLE3.We have to go
 in to these grouping since we need drilled down info of roles per user, per
 service provider etc.

 So when it comes to user interactions , as per[1] he can click on per
 user login auth success table. According to the clicked user above Area
 chart, and all other horizontal bar charts should be changed for that user.
 

Re: [Architecture] Security Authentication Analytics

[Sajith Ravindra]

Hi Damith,

Do we support these statistics per tenant? So that each tenant can view
these statistics per tenant.

Thanks
*,Sajith Ravindra*
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware

mobile: +94 77 2273550
blog: http://sajithr.blogspot.com/


On Mon, Mar 28, 2016 at 4:08 AM, Damith Wickramasinghe 
wrote:

> Sure.We will schedule.
>
> Regards,
> Damith.
>
> On Mon, Mar 28, 2016 at 4:02 PM, Srinath Perera  wrote:
>
>> As per our chat, lets schedule a review.
>>
>> --Srinath
>>
>> On Sun, Mar 27, 2016 at 11:19 PM, Damith Wickramasinghe > > wrote:
>>
>>> Hi All,
>>>
>>> When it comes to Security analytics story we have broken it down to
>>> mainly three sections(Can be more in the future). Namely Authentication
>>> Analytics , Authorization Analytics and Audit trail for identity
>>> artifacts.  As for the first phase we are focusing on Authentication
>>> Analytics and Audit trail for identity artifacts.
>>>
>>> Data Summarization
>>>
>>> For Authentication Analytics when a user authenticates we are publishing
>>> related data from IS side for a raw table which will get persisted In DAS.
>>> Then using scheduled spark scripts we are summarizing the data to
>>> yearly,monthly,daily,hourly,minutely and secondly tables. (We may drop the
>>> secondly table since it may contain lot of data and will be not efficient
>>> when carrying out aggregate operations. Also these scheduled spark scripts
>>> will run incrementally without summarizing previous data again.)
>>>
>>> As per the [1] we have following charts.We are only considering login
>>> success and failure scenarios. (As discussed with IS team we dropped logout
>>> success and failure scenarios for now.Since showing those statistics are
>>> not much important.)
>>>
>>>- overall authentication success and failure count during a time
>>>range - Area chart
>>>- per user Authentication success count - horizontal bar chart
>>>- per user Authentication failure count - horizontal bar chart
>>>- per role Authentication success count - horizontal bar chart
>>>- per role Authentication failure count - horizontal bar chart
>>>
>>>  etc. As above there are charts for service provider,identity
>>> provider,region and for ip ranges as well.
>>>
>>> Above Area chart for overall authentication success and failure count
>>> can be further drilled down as user clicks on horizontal bar charts(per
>>> user,per role etc). To cater this we have a one table structure with
>>> columns (if we take monthly table) Per month -> Per user -> Per roles ->
>>> Per serviceProvider -> Per identityProvider -> Per region -> Per Ip ->
>>> authSuccessCount and authFailureCount. Ill call this TABLE1.
>>>
>>> Please note in above table we are keeping the roles as comma separated
>>> values. so when a role is clicked we can identify authSuccess and
>>> authFailure count using DAS score function[2]. We discussed on getting this
>>> comma separated roles per event from IS directly since it will make things
>>> easier when it comes to summarization logic.
>>>
>>> For all other horizontal bar charts except role table , we are following
>>> below table structure.
>>>
>>> (If we take monthly table for user) Per month -> Per user
>>> ->authSuccessCount and authFailureCount . Ill call this TABLE2.
>>>
>>> Its true that these information can also be achieved from TABLE1. But
>>> since it has multi level grouped data, data aggregation will take more
>>> time. So having on level grouping will allow less records to be aggregated.
>>>
>>> For the role table we need a row for each role with corresponding
>>> authSuccessCount and authFailureCount. But as mentioned above since we are
>>> sending roles as comma separated values we do not have a efficient way to
>>> separate each role and construct the table. So we thought of getting the
>>> data as duplicated events(per user will have multiple roles so a event will
>>> be duplicated because of the role) from IS side and do the summarization.
>>>
>>> (If we take monthly table per role) Per Role -> Per User -> Per Service
>>> provider -> Per identity Provider -> Per region -> Per IP ->
>>> authSuccessCount and authFailureCount. Ill call this TABLE3.We have to go
>>> in to these grouping since we need drilled down info of roles per user, per
>>> service provider etc.
>>>
>>> So when it comes to user interactions , as per[1] he can click on per
>>> user login auth success table. According to the clicked user above Area
>>> chart, and all other horizontal bar charts should be changed for that user.
>>> So if I take service provider auth success and failure charts they first
>>> will (Before user click) will generate the chart from TABLE2 and (after
>>> user clicked) will generate the data from TABLE1 which is service providers
>>> for that username for the given time range.
>>>
>>> But for role since all 

Re: [Architecture] Security Authentication Analytics

[Damith Wickramasinghe]

Sure.We will schedule.

Regards,
Damith.

On Mon, Mar 28, 2016 at 4:02 PM, Srinath Perera  wrote:

> As per our chat, lets schedule a review.
>
> --Srinath
>
> On Sun, Mar 27, 2016 at 11:19 PM, Damith Wickramasinghe 
> wrote:
>
>> Hi All,
>>
>> When it comes to Security analytics story we have broken it down to
>> mainly three sections(Can be more in the future). Namely Authentication
>> Analytics , Authorization Analytics and Audit trail for identity
>> artifacts.  As for the first phase we are focusing on Authentication
>> Analytics and Audit trail for identity artifacts.
>>
>> Data Summarization
>>
>> For Authentication Analytics when a user authenticates we are publishing
>> related data from IS side for a raw table which will get persisted In DAS.
>> Then using scheduled spark scripts we are summarizing the data to
>> yearly,monthly,daily,hourly,minutely and secondly tables. (We may drop the
>> secondly table since it may contain lot of data and will be not efficient
>> when carrying out aggregate operations. Also these scheduled spark scripts
>> will run incrementally without summarizing previous data again.)
>>
>> As per the [1] we have following charts.We are only considering login
>> success and failure scenarios. (As discussed with IS team we dropped logout
>> success and failure scenarios for now.Since showing those statistics are
>> not much important.)
>>
>>- overall authentication success and failure count during a time
>>range - Area chart
>>- per user Authentication success count - horizontal bar chart
>>- per user Authentication failure count - horizontal bar chart
>>- per role Authentication success count - horizontal bar chart
>>- per role Authentication failure count - horizontal bar chart
>>
>>  etc. As above there are charts for service provider,identity
>> provider,region and for ip ranges as well.
>>
>> Above Area chart for overall authentication success and failure count can
>> be further drilled down as user clicks on horizontal bar charts(per
>> user,per role etc). To cater this we have a one table structure with
>> columns (if we take monthly table) Per month -> Per user -> Per roles ->
>> Per serviceProvider -> Per identityProvider -> Per region -> Per Ip ->
>> authSuccessCount and authFailureCount. Ill call this TABLE1.
>>
>> Please note in above table we are keeping the roles as comma separated
>> values. so when a role is clicked we can identify authSuccess and
>> authFailure count using DAS score function[2]. We discussed on getting this
>> comma separated roles per event from IS directly since it will make things
>> easier when it comes to summarization logic.
>>
>> For all other horizontal bar charts except role table , we are following
>> below table structure.
>>
>> (If we take monthly table for user) Per month -> Per user
>> ->authSuccessCount and authFailureCount . Ill call this TABLE2.
>>
>> Its true that these information can also be achieved from TABLE1. But
>> since it has multi level grouped data, data aggregation will take more
>> time. So having on level grouping will allow less records to be aggregated.
>>
>> For the role table we need a row for each role with corresponding
>> authSuccessCount and authFailureCount. But as mentioned above since we are
>> sending roles as comma separated values we do not have a efficient way to
>> separate each role and construct the table. So we thought of getting the
>> data as duplicated events(per user will have multiple roles so a event will
>> be duplicated because of the role) from IS side and do the summarization.
>>
>> (If we take monthly table per role) Per Role -> Per User -> Per Service
>> provider -> Per identity Provider -> Per region -> Per IP ->
>> authSuccessCount and authFailureCount. Ill call this TABLE3.We have to go
>> in to these grouping since we need drilled down info of roles per user, per
>> service provider etc.
>>
>> So when it comes to user interactions , as per[1] he can click on per
>> user login auth success table. According to the clicked user above Area
>> chart, and all other horizontal bar charts should be changed for that user.
>> So if I take service provider auth success and failure charts they first
>> will (Before user click) will generate the chart from TABLE2 and (after
>> user clicked) will generate the data from TABLE1 which is service providers
>> for that username for the given time range.
>>
>> But for role since all informations exist in TABLE3 we can retrieve roles
>> per specific user from it.
>>
>> Above is the basic table structure and there will be 30 table for now.
>>
>> Any suggestions and thoughts are highly appreciated.
>>
>> [1]
>> https://docs.google.com/document/d/1gJUqkUE3oyxguipr9nAzM31TlB7lcmSH2ezYOM-7KIA/edit
>> [2]
>> https://docs.wso2.com/display/DAS301/Drilling+Down+Through+Categories+via+JS+API
>>
>> Regards,
>> Damith.
>>
>> --
>> Software Engineer
>> WSO2 Inc.; http://wso2.com
>> 

Re: [Architecture] Security Authentication Analytics

[Srinath Perera]

As per our chat, lets schedule a review.

--Srinath

On Sun, Mar 27, 2016 at 11:19 PM, Damith Wickramasinghe 
wrote:

> Hi All,
>
> When it comes to Security analytics story we have broken it down to mainly
> three sections(Can be more in the future). Namely Authentication Analytics
> , Authorization Analytics and Audit trail for identity artifacts.  As for
> the first phase we are focusing on Authentication Analytics and Audit trail
> for identity artifacts.
>
> Data Summarization
>
> For Authentication Analytics when a user authenticates we are publishing
> related data from IS side for a raw table which will get persisted In DAS.
> Then using scheduled spark scripts we are summarizing the data to
> yearly,monthly,daily,hourly,minutely and secondly tables. (We may drop the
> secondly table since it may contain lot of data and will be not efficient
> when carrying out aggregate operations. Also these scheduled spark scripts
> will run incrementally without summarizing previous data again.)
>
> As per the [1] we have following charts.We are only considering login
> success and failure scenarios. (As discussed with IS team we dropped logout
> success and failure scenarios for now.Since showing those statistics are
> not much important.)
>
>- overall authentication success and failure count during a time range
>- Area chart
>- per user Authentication success count - horizontal bar chart
>- per user Authentication failure count - horizontal bar chart
>- per role Authentication success count - horizontal bar chart
>- per role Authentication failure count - horizontal bar chart
>
>  etc. As above there are charts for service provider,identity
> provider,region and for ip ranges as well.
>
> Above Area chart for overall authentication success and failure count can
> be further drilled down as user clicks on horizontal bar charts(per
> user,per role etc). To cater this we have a one table structure with
> columns (if we take monthly table) Per month -> Per user -> Per roles ->
> Per serviceProvider -> Per identityProvider -> Per region -> Per Ip ->
> authSuccessCount and authFailureCount. Ill call this TABLE1.
>
> Please note in above table we are keeping the roles as comma separated
> values. so when a role is clicked we can identify authSuccess and
> authFailure count using DAS score function[2]. We discussed on getting this
> comma separated roles per event from IS directly since it will make things
> easier when it comes to summarization logic.
>
> For all other horizontal bar charts except role table , we are following
> below table structure.
>
> (If we take monthly table for user) Per month -> Per user
> ->authSuccessCount and authFailureCount . Ill call this TABLE2.
>
> Its true that these information can also be achieved from TABLE1. But
> since it has multi level grouped data, data aggregation will take more
> time. So having on level grouping will allow less records to be aggregated.
>
> For the role table we need a row for each role with corresponding
> authSuccessCount and authFailureCount. But as mentioned above since we are
> sending roles as comma separated values we do not have a efficient way to
> separate each role and construct the table. So we thought of getting the
> data as duplicated events(per user will have multiple roles so a event will
> be duplicated because of the role) from IS side and do the summarization.
>
> (If we take monthly table per role) Per Role -> Per User -> Per Service
> provider -> Per identity Provider -> Per region -> Per IP ->
> authSuccessCount and authFailureCount. Ill call this TABLE3.We have to go
> in to these grouping since we need drilled down info of roles per user, per
> service provider etc.
>
> So when it comes to user interactions , as per[1] he can click on per user
> login auth success table. According to the clicked user above Area chart,
> and all other horizontal bar charts should be changed for that user. So if
> I take service provider auth success and failure charts they first will
> (Before user click) will generate the chart from TABLE2 and (after user
> clicked) will generate the data from TABLE1 which is service providers for
> that username for the given time range.
>
> But for role since all informations exist in TABLE3 we can retrieve roles
> per specific user from it.
>
> Above is the basic table structure and there will be 30 table for now.
>
> Any suggestions and thoughts are highly appreciated.
>
> [1]
> https://docs.google.com/document/d/1gJUqkUE3oyxguipr9nAzM31TlB7lcmSH2ezYOM-7KIA/edit
> [2]
> https://docs.wso2.com/display/DAS301/Drilling+Down+Through+Categories+via+JS+API
>
> Regards,
> Damith.
>
> --
> Software Engineer
> WSO2 Inc.; http://wso2.com
> 
> lean.enterprise.middleware
>
> mobile: *+94728671315 <%2B94728671315>*
>
>
> ___
> Architecture mailing