Re: [Architecture] [APIM][Micro-Gateway][Analytics] Analytics for Micro-gateway

[Sriskandarajah Suhothayan]

IMO, we can upload the files to any of the DAS nodes and let it share among
its HA seems to be the correct approach.
Here we can set the event duplicated in the cluster property as falls all
the DAS receivers.
Doing so events will be shared and processed on both DAS HA nodes.

On Tue, Jul 3, 2018 at 10:18 AM Fazlan Nazeem  wrote:

> Hi Sajith,
>
> The Gateway will not have access to any of the databases. Therefore it
> will use the filesystem to temporarily store the zip file until it is
> uploaded to the Analytics server.
>
> On Tue, Jul 3, 2018 at 8:24 AM Sajith Perera  wrote:
>
>>
>> Whether we buffering the zip file in micro GW if the analytics server not
>> available? In such cases what if we give an option to persist zip file
>> directly to database by micro GW as analytics server anyway reading the
>> data from database.
>>
>> Regards,
>> SajithD
>>
>> On Tue, Jul 3, 2018 at 2:47 AM Tishan Dahanayakage 
>> wrote:
>>
>>> Sorry for the delayed reply Nuwan. I was traveling.
>>>
>>> On Fri, Jun 29, 2018 at 9:19 AM, Nuwan Dias  wrote:
>>>


 On Fri, Jun 29, 2018 at 4:43 PM Tishan Dahanayakage 
 wrote:

> Hi Dinusha,
>
> On Fri, Jun 29, 2018, 4:37 PM Dinusha Dissanayake 
> wrote:
>
>> Hi Tishan,
>>
>>>
> One more thing. Can't we just save these zip files to file system
> rather than stressing STATS_DB. We use STATS_DB mainly to store end
> analytics data which is used by presentation layer(Dashboards). WDYT?
>
 This would be problematic in HA deployment. If we keep them in the
 file system and if a node goes down, we won't be able to retrieve  the
 event data in files in that node.

>>> ​That we can solve by publishing to both DAS nodes from GW. Even
>>> earlier I was discussing with Fazlan to avoid adding file to DB by using
>>> file tail adaptor but later reverted due to zip files. But given that we
>>> are now using custom adaptor we can use files :)
>>>
>> If we publish to both DAS nodes, then the files would be available in
>> both nodes. When event publishing is happening by reading those files, 
>> the
>> same file will be processed from both the nodes right? :)
>> Then the same events will be accumulated twice as I see.
>>
> No that is handled by HA implementation.
>

 Didn't get that part. What do you mean by "handled by HA
 implementation"?

>>>
>>> ​When we configure DAS in HA mode, both receivers can receive same event
>>> and yet active node will do the presentation part.​
>>>

 Another question is, how does the gateways know the number of DAS nodes
 to upload to? In a HA scenario, the gateway will only see the LB URL
 (because DAS will be proxied via an LB). In that case the gateway only
 uploads to the LB url, it has no idea how many DAS nodes are behind that LB
 and it doesn't need to know as well.

>>> ​Yeah if DAS is proxied via LB then publishing to both nodes is an
>>> issue. Then file based solution becomes obsolete as we can't share files
>>> in-between nodes
>>>
>>> /Tishan
>>>

 To me it sounds like the problems we may have to solve by persisting to
 local file systems in each DAS node are much more severe than the overhead
 that gets added to the DB. Because in reality, each gateway will only
 upload these files like once every 15 minutes. So in a system with 1
 gateway, we're introducing just 1 additional DB read/write per every 15
 minutes. Yes, it increases with the number of gateways in the system, in
 which case we may have to reduce the upload frequency.



> /Tishan
>
>>
>>> /Tishan
>>>
 ​
 ​

>>>
>
 /Tishan
>
> On Fri, Jun 29, 2018 at 2:42 PM, Tishan Dahanayakage <
> tis...@wso2.com> wrote:
>
>> Fazlan,
>>
>> On Fri, Jun 29, 2018 at 2:17 PM, Fazlan Nazeem 
>> wrote:
>>
>>> Hi all,
>>>
>>> At the moment, analytics for microgateway is supported via a
>>> JAX-RS web app and a custom component which are deployed in APIM 
>>> publisher
>>> node. The component was responsible for publishing the analytics 
>>> data
>>> persisted in a DB table to the Analytics server via thrift. As an
>>> improvement for this, we have planned to move the web app to 
>>> Analytics
>>> server and process the events within itself which will remove the 
>>> overhead
>>> of publishing data via thrift. The micro-gateways will then upload 
>>> the zip
>>> files with analytics data directly to the analytics server so that 
>>> we can
>>> eliminate an unnecessary network hop.
>>>
>>> For this, we have developed a working prototype which follows
>>> the following 

Re: [Architecture] SAML Artifact Binding - Server Side Implementations

[Maduranga Siriwardena]

Databases can handle large text fields. Column type depends on the
database. For example [1] shows few MySql column types that can handle
large texts.

And in the same time there are some database column types that can handle
xml etc. You will need to do some research to to find suitable column type
for your requirement.

[1]
https://stackoverflow.com/questions/6766781/maximum-length-for-mysql-type-text

Thanks,

On Tue, Jul 3, 2018 at 12:26 PM Vihanga Liyanage  wrote:

> Hi Farasath,
>
>>
>> SAML Assertion size is going to depend with the number of requested
>> claims, signing, encryption etc. How are we planning to handle this
>> ​?
>>
>
> ​That is a valid question! The initial value, 4096 was used in the
> IDN_SAML2_ASSERTION_STORE table. But with my implementation, later I found
> out that it's not enough and used 5120 for now. ​Is there a maximum size
> that we can decide on?
>
>> ​
>>
>>
>

-- 
Maduranga Siriwardena
Senior Software Engineer
WSO2 Inc; http://wso2.com/

Email: madura...@wso2.com
Mobile: +94718990591
Blog: *https://madurangasiriwardena.wordpress.com/
*

___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] SAML Artifact Binding - Server Side Implementations

[Vihanga Liyanage]

Hi Farasath,

>
> SAML Assertion size is going to depend with the number of requested
> claims, signing, encryption etc. How are we planning to handle this
> ​?
>

​That is a valid question! The initial value, 4096 was used in the
IDN_SAML2_ASSERTION_STORE table. But with my implementation, later I found
out that it's not enough and used 5120 for now. ​Is there a maximum size
that we can decide on?

> ​
>
>
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] SAML Artifact Binding - Server Side Implementations

[Farasath Ahamed]

On Mon, Jul 2, 2018 at 2:48 PM, Vihanga Liyanage  wrote:

> Hi all,
>
> As for the discussion we had earlier [1], here attached the initial table
> design to store the SAML assertions against the artifact ID. Please let me
> know your concerns regarding this or anything discussed earlier.
>
> [image: image.png]
>

SAML Assertion size is going to depend with the number of requested claims,
signing, encryption etc. How are we planning to handle this?

>
> [1] - "Invitation: [Architecture Review] - SAML Artifact Binding" @
> engineering
>
> On Fri, Jun 22, 2018 at 4:38 PM Vihanga Liyanage  wrote:
>
>> [+ Dev]
>>
>> On Fri, Jun 22, 2018 at 3:23 PM Vihanga Liyanage 
>> wrote:
>>
>>> Hi all,
>>>
>>> As I'm going through the specifications, I came across following
>>> problems.
>>>
>>>- The above diagram shows Login Response binding with SAML art.
>>>There are other aspects of this as well such as Login Request Binding,
>>>Logout Request Binding, etc. Below diagram shows both login request and
>>>response bound with SAML art.
>>>
>>> [image: image.png]
>>>Question is, which should we do first. I think login
>>> response binding is better, to begin with.
>>>
>>>- IDP has to store a reference to the artifact so that it can
>>>respond with the SAML assertion to the SP. We can either generate the
>>>assertion and store it completely in the database and send that, or we 
>>> can
>>>generate and send the assertion once the artifact is received. What 
>>> should
>>>be the best method?
>>>- Should we keep a status about the artifact and assertion in our
>>>DB? If yes, what are the use cases?
>>>- What should happen if an artifact is sent again by someone after
>>>the assertion is issued? The spec says the following but I didn't see any
>>>specific instruction on what to do.
>>>- *"It is RECOMMENDED that artifact receivers also enforce a
>>>   single-use semantic on the artifact values they receive, to prevent an
>>>   attacker from interfering with the resolution of an artifact by a user
>>>   agent and then re-submitting it to the artifact receiver. If an 
>>> attempt to
>>>   resolve an artifact does not complete successfully, the artifact 
>>> SHOULD be
>>>   placed into a blocked artifact list for a period of time that exceeds 
>>> a
>>>   reasonable acceptance period during which the artifact issuer would 
>>> resolve
>>>   the artifact."*
>>>
>>> Please let me know your thoughts on the above.
>>>
>>> Regards,
>>> Vihanga.
>>>
>>> On Wed, Jun 20, 2018 at 10:28 AM Vihanga Liyanage 
>>> wrote:
>>>
 Hi all,

 I've started working on the server-side implementation of SAML Artifact
 Binding. The basic idea is as follows.

 When authentication is done via SAML, SAML assertion is sent to the
 user agent (browser) as a direct response from the IDP. One disadvantage of
 this method is the possibility of communication messages being intersepted
 at the browser. Also, there could be limitations on browsers such as limits
 on query string / POST payload sizes, no support for JavaScript, etc. To
 overcome these problems, SAML Artifact Binding has been introduced.

 When the user is authenticated, the IDP responds with a key known as
 SAMLart, which will be then sent to the service provider by the browser.
 Then the SP uses this key to request the actual SAML assertion from the IDP
 via a back channel call. This method reduces the use of browsers compared
 to the old method. Below diagram shows the request flow with SAML Artifact
 Binding.

 [image: image.png]

 ​Currently the client side implementations have been completed and
 discussed here [1]. The goal of this project is to implement the necessary
 backend components following the official SAML specification [2]
 
 .

 I highly appriciate your valuable concerns and input on this.

 Best regards,
 Vihanga.

 [1] - "[Architecture] [IAM] SAML Artifact Binding" @
 architecture@wso2.org
 [2] - https://www.oasis-open.org/committees/download.php/35387/
 sstc-saml-bindings-errata-2.0-wd-05-diff.pdf
 
 --

 Vihanga Liyanage

 Software Engineer | WS*O₂* Inc.

 M : +*94710124103* | http://wso2.com

 [image: http://wso2.com/signature] 

>>>
>>>
>>> --
>>>
>>> Vihanga Liyanage
>>>
>>> Software Engineer | WS*O₂* Inc.
>>>
>>> M : +*94710124103* | http://wso2.com
>>>
>>> [image: http://wso2.com/signature] 
>>>
>>
>>
>> --
>>
>> Vihanga Liyanage
>>
>>