Re: [Architecture] File Based Database for Message Broker Store

[Wishmitha Mendis]

Hi Kevin,

I just tried jedis for the initial implementation, no actual reason reason
to be honest. I will try other java-clients you mentioned, and figure out
what will fit the best for the project scenario.

Best Regards,

On Fri, Jul 14, 2017 at 11:17 PM, Kevin Ratnasekera  wrote:

> Hi Wishmitha,
>
> There are many java clients available for redis, there are several very
> good clients, see comparison [1]. Any reason using jedis over others?
>
> [1] https://redis.io/clients#java
>
> Regards
> Kevin
>
>
>
> On Fri, Jul 14, 2017 at 11:02 PM, Wishmitha Mendis 
> wrote:
>
>> Hi Indika,
>>
>> I have started a basic implementation to test the concurrency handling,
>> message order and transaction support of Redis. Basic message producers and
>> consumers were implemented and messages are stored in a basic Redis queue.
>> GitHub : https://github.com/Wishmitha/RedisTesting
>>
>> Best Regards,
>>
>> On Thu, Jul 13, 2017 at 6:51 PM, Indika Sampath  wrote:
>>
>>> Hi Wishmitha,
>>>
>>> No SQL database is not an option as we have prior experience with
>>> Cassandra. Basically, the important part is to find out performance and
>>> fault tolerance capability in the database. Message broker
>>> read/write/delete records all the time and database should be able to
>>> handle high concurrency. Some metadata and message content write as byte
>>> data. Also, message order matters when delivery. Hence database should
>>> capable to sort records rather than bringing that logic into broker code.
>>> As you mentioned transaction is another factor we are looking in the
>>> database. Shall we start POC with Redis and see how it goes?
>>>
>>> Cheers!
>>>
>>> On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis 
>>> wrote:
>>>
 Hi all,

 I am working on a project to replace the current RDBMS based database
 of the message broker store with a file based database system. I have been
 researching on potential file based databases which can be used in the
 project scenario. I have evaluated the pros and cons of the each of the
 potential databases mainly based on read and write performances,
 transaction support, data structure (queue) implementation support and
 overhead of replacement.

 A summary of database evaluation is represented in the following table.


 Database

 Pros

 Cons

 LevelDB

-

A simple key value storage
-

Used in ActiveMQ (replaced with KahaDB)


1.

Fast due to sorted keys
2.

Simple transaction support with batch( ) operation
3.

Sublevel feature (provides more organization and transactions
facilitation)
4.

LiveStream feature (able to query things which are yet being
receiving)
5.

Support triggers, groupby (mapreduce) and join (fanout) etc.


1.

Performance decrease in concurrent / multi threaded scenarios.
(improved in RocksDB.)
2.

No object relational mapping.
3.

Poor performance when memory is not enough.

 MongoDB


-

Document oriented
-

Can define JSON schemas



1.

Direct object mapping (good replacement for RDBMS)
2.

Can implement queue like data structures (store messages in
separate documents and delete them once delivered / already implemented
data structures : https://www.npmjs.com/package/mongodb-queue)
3.

Capped collections (automatically remove old docs to make space for
new docs.)


1.

No transaction support for multiple documents.
2.

Low performance compared to other NoSQL databases.

 Caassandra


-

Column oriented (a key is denoted by row and an attribute is
denoted by a column which can dynamically change.)


-

Node - Cluster configuration (all nodes are identical)


1.

High availability (No single point of failure. In case of node
failure other identical nodes are available.)
2.

No data loss due to replication (identical nodes)
3.

Easy to model (column store is very similar to RDBMS tables.)
4.

Similar query language to SQL (CQL)
5.

High performance (No master node concept. Any node can provide
requested query. No performance bottleneck.)
6.

Transaction implementation


1.

Does not support queue structure (message queue implementation is
an anti-pattern : http://www.datastax.com/dev/bl

Re: [Architecture] [CARBON] C4 Migration from Log4J 1.x to Log4J 2.x

[Ruwan Abeykoon]

Hi Irunika,
In C4, applications use commons logging. We never wanted to use Log4J
directly. Hence the architecture you proposed is already there in C4.
In C5 we use slf4j, in place of commons logging, which architecturally the
same.

Hence I do not see any value in introducing another WSO2 specific logging
framework/wrapper/etc.

For the original ${subject}, would not it work simply using log4j2 library,
removing all log4j 1 library with change in log4j.property file
appropriately?
Yes, it needs to have a change in Carbon**Appenders to use log4j2 API, but
this is only at Carbon-Logging module. I guess we can have new version with
log4j2.
In short, IMO, there will be no change in applications, but may be little
change in kernel implementation(which is transparent to apps) in log4j2
migration.

Cheers,
Ruwan


On Fri, Jul 14, 2017 at 3:37 PM, Irunika Weeraratne 
wrote:

> Hi Asma,
>>
>> The following approaches were found and evaluated to identify the most
>> suitable one.
>>
>>1. Using pax-logging
>>- The pax-logging api and the pax-logging-log4j bridge will have to
>>   be used.
>>   - Supposedly, pax-logging will feed the front-end logging into its
>>   log4j back-end using its built-in adapter which we are trying to
>>   achieve at the moment
>>2. Using Log4J 2.x
>>   - This will need the log4j2-core, log4j2-api and the bridges to
>>   map the front-end logging api to Log4J 2.x back-end
>>   - Also this needs two OSGi fragments to customize the log4j-core
>>   and log4j-api work in OSGi
>>   - This requires the log4j2.properties file to be in the fragment
>>   used with log4j-core and therefore, the file cannot be edited later
>>   - Also, external .jar files are not able to find the logging
>>   implementation which we are still investigating as to why this happens
>>3. Using both Log4j 1.x and 2.x
>>   - While C4 will use the latter as the back-end, the external jars
>>   needing v1 will use the former
>>   - This could lead to threading issues since appenders from two
>>   log4J versions will write to a single file simultaneously
>>4. Using the Log4J 1.x bridge [4]
>>   - The Log4J 1.x jar is replaced with the 1.x bridge api and the
>>   required log4j2 files. This will redirect all the logs from Log4j 1.x 
>> to 2.x
>>   - This only works if there are no custom appenders
>>
>> IMHO all these approaches also have a common issue. When we try to use
> any of those APIs we have to have to change our code base again and
> again according to the API we are using.
>
> IMO what we should do is write our own API for Logging and in the
> implementation, we can use any of those as the backend. So only the
> implementation will be changed while our code base remains same.
>
> Please find the diagram below.
>
> [image: Inline image 1]
>
> WDYT?
>
> Thanks and best regards,
> Irunika
>
>
>
> *Irunika Weeraratne*
> *Software Engineer | WSO2, Inc. *
> *Email : irun...@wso2.com *
> *LinkedIn : https://lk.linkedin.com/in/irunika
> *
> *Mobile : +94712403314 <+94%2071%20240%203314>*
> *Lean . Enterprise . Middleware*
>
>
> On Wed, Jul 12, 2017 at 7:53 PM, Asma Jabir  wrote:
>
>> Hi
>>
>> We have been looking into $subject and these are the findings we have
>> arrived at so far.
>>
>> Carbon 4.x.x currently uses Log4J 1.x as the logging back-end and
>> following will have to be done when migrating to Log4J 2.x.
>>
>>1. Rewriting custom appenders
>>- Current Log4J 1.x method - Rewriting of Logging Events by extending
>>   appropriate classes (i.e.: 
>> AppenderSkeleton/ConsoleAppender/DailyRollingFileAppender)
>>   [1]
>>   - Proposed Log4J 2.x method - Use of plugins to convert the log
>>   record into a required pattern with custom fields [2]
>>2. Replace log4j.properties with a log4j2.properties file
>>   - The properties file should be changed to log4j2.properties and
>>   the syntax will different from v1 format [3]
>>3. Remove
>>   - Direct use of log4j in org.wso2.carbon.server.Main - used to
>>   remove all appenders added in the non-OSGi environment
>>   - Package org.wso2.carbon.utils.logging.appenders as this should
>>   be rewritten as converter plugins
>>
>>
>> The following approaches were found and evaluated to identify the most
>> suitable one.
>>
>>1. Using pax-logging
>>- The pax-logging api and the pax-logging-log4j bridge will have to
>>   be used.
>>   - Supposedly, pax-logging will feed the front-end logging into its
>>   log4j back-end using its built-in adapter which we are trying to
>>   achieve at the moment
>>2. Using Log4J 2.x
>>   - This will need the log4j2-core, log4j2-api and the bridges to
>>   map the front-end logging api to Log4J 2.x back-end
>>   - Also this needs two OSGi fragments to 

Re: [Architecture] IS Authentication Flow Behavior Between Multiple Service Providers

[Johann Nallathamby]

On Sat, Jul 15, 2017 at 5:55 AM Darshana Gunawardana 
wrote:

> Hi Johann,
>
> On Fri, Jul 14, 2017 at 5:45 PM Johann Nallathamby 
> wrote:
>
>> Can we change the implementation as follows:
>>
>> If user is going to login to application Y, which has steps 1 to *m*,
>> and user has already logged into Y
>>
> Should be corrected as application X?
>

Yes :)


> which has steps 1 to *n*,
>>
> given p <= m , n
>> User is already authenticated to step *p* of application Y, if user has
>> logged in from any of the optional IdPs in step *p*, within the sequence
>> of application X, between steps *p* and n*.*
>>
>
> Anyway I don't think above algorithm is correct because it base on number
> of steps.
>
> 1. The simple way of fixing this is handle LOCAL idp specially.
>
> Currently the framework calculates to bypass the step p (when loging to
> app Y), if user have already logged in at least one of optional IdP in
> previous login (app X).
>
> We have to add extra level of validation if the previously authenticated
> IdP picked as the LOCAL IdP where it should check the whether previously
> authenticated authenticator (basic authenticator) and current available
> authenticator (TOTP) same or not.
>
> If it's not same, prompt for authentication without bypassing.
>

+1. This is also a good idea.

>
> 2. The generic way of fixing this is, consider ACR for each IdP
>
> Extending the previous solution, (rather having special checks for LOCAL
> IdP), check on the authentication level / ACR of the  any IdP when it
> decide whether user already logged in with that IdP.
>

I believe this is the generic solution coming with IS 5.5.0.


> WDYT?
>
> Regards,
>
>
>> Can we do something like this for 5.4.0? IMO this will be more practical
>> and step up authentication will work seamlessly.
>>
>> Regards,
>> Johann.
>>
>> On Fri, Jul 14, 2017 at 10:58 AM, Ishara Karunarathna 
>> wrote:
>>
>>> Hi Johan,
>>>
>>> On Fri, Jul 14, 2017 at 1:56 PM, Johann Nallathamby 
>>> wrote:
>>>
 Hi Asela,

 On Fri, Jul 14, 2017 at 9:34 AM, Asela Pathberiya 
 wrote:

>
>
> On Fri, Jul 14, 2017 at 11:31 AM, Harsha Kumara 
> wrote:
>
>> Hi All,
>>
>> This is regarding the behavior of Authentication flow between
>> multiple service providers.
>>
>> I have created two service providers with following configurations.
>>
>> *SP1*
>>
>> This service provider has two options which allow to users to login
>> either Basic Authentication scheme or Facebook
>>
>> Configuration
>>
>> 1 Authentication Step with MultiOption with Basic Auth and Facebook.
>>
>>
>> *SP2*
>>
>> This service provider has two authentication steps which allow to
>> users to login either Basic Authentication scheme or Facebook and second
>> authentication step with TOTP.
>>
>> Configuration
>>
>> 2 Authentication Steps
>>
>>- 1 Authentication Step with MultiOption with Basic Auth and
>>Facebook.
>>- 2 Authentication Step with TOTP
>>
>>
>> *Behavioral Concern*
>>
>> I have configured two applications with SP1 and SP2 respectively.
>> Then I have logged into the first application with Basic Authentication
>> Scheme which is configured in SP1.
>>
>> But when I going to authentication with my second application which
>> configured with SP2, I have logged into it automatically.
>>
>> Shouldn't it ask for TOTP authentication? Because first application
>> only authenticated with Basic Auth but my second application required 
>> Basic
>> Auth + TOTP.
>>
>
> Yes. It should...  Session contains the authenticated SP details.
> Therefore;  it can decide based on the SP...  If it is not working,  it
> seems like a bug..
>

 I don't think it has ever worked like that because we maintain a list
 of IdPs against which we have authenticated so far for any service
 provider. We don't consider the step number here. So since basic auth and
 TOTP both are for LOCAL IdP, IS considers SP2 also authenticated if we
 authenticate with username/password only for SP1.

 We were actually trying this as a logical workaround to have step up
 authentication so when the application decided it needs 2nd factor
 authentication to send a request using the new service provider ID, but it
 seems not possible with current version of IS.

>>> I tested this and got the same behavior.
>>> As I remember we have authenticator information as well in the context
>>> so we should be able to fix this in future.
>>>
>>> -Ishara
>>>

 Regards,
 Johann.


> Thanks,
> Asela.
>
>
>>
>> Thanks,
>> Harsha
>>
>> --
>> Harsha Kumara
>> Software Engineer, WSO2 Inc.
>> 

Re: [Architecture] IS Authentication Flow Behavior Between Multiple Service Providers

[Farasath Ahamed]

Farasath Ahamed
Software Engineer, WSO2 Inc.; http://wso2.com
Mobile: +94777603866
Blog: blog.farazath.com
Twitter: @farazath619 




On Sat, Jul 15, 2017 at 11:35 AM, Johann Nallathamby 
wrote:

>
>
> On Sat, Jul 15, 2017 at 5:55 AM Darshana Gunawardana 
> wrote:
>
>> Hi Johann,
>>
>> On Fri, Jul 14, 2017 at 5:45 PM Johann Nallathamby 
>> wrote:
>>
>>> Can we change the implementation as follows:
>>>
>>> If user is going to login to application Y, which has steps 1 to *m*,
>>> and user has already logged into Y
>>>
>> Should be corrected as application X?
>>
>
> Yes :)
>
>
>> which has steps 1 to *n*,
>>>
>> given p <= m , n
>>> User is already authenticated to step *p* of application Y, if user has
>>> logged in from any of the optional IdPs in step *p*, within the
>>> sequence of application X, between steps *p* and n*.*
>>>
>>
>> Anyway I don't think above algorithm is correct because it base on number
>> of steps.
>>
>> 1. The simple way of fixing this is handle LOCAL idp specially.
>>
>> Currently the framework calculates to bypass the step p (when loging to
>> app Y), if user have already logged in at least one of optional IdP in
>> previous login (app X).
>>
>> We have to add extra level of validation if the previously authenticated
>> IdP picked as the LOCAL IdP where it should check the whether previously
>> authenticated authenticator (basic authenticator) and current available
>> authenticator (TOTP) same or not.
>>
>> If it's not same, prompt for authentication without bypassing.
>>
>
> +1. This is also a good idea.
>

Shouldn't we do the same for Federated IDPs as well?
I mean we can configure multiple federated authenticators within the same
Fed IDP and then use each authenticator in different steps. This would be a
similar situation to the LOCAL IDP case right?


>
>
>> 2. The generic way of fixing this is, consider ACR for each IdP
>>
>> Extending the previous solution, (rather having special checks for LOCAL
>> IdP), check on the authentication level / ACR of the  any IdP when it
>> decide whether user already logged in with that IdP.
>>
>
> I believe that is the solution coming with IS 5.5.0.
>
>>
>> WDYT?
>>
>> Regards,
>>
>>
>>> Can we do something like this for 5.4.0? IMO this will be more practical
>>> and step up authentication will work seamlessly.
>>>
>>> Regards,
>>> Johann.
>>>
>>> On Fri, Jul 14, 2017 at 10:58 AM, Ishara Karunarathna 
>>> wrote:
>>>
 Hi Johan,

 On Fri, Jul 14, 2017 at 1:56 PM, Johann Nallathamby 
 wrote:

> Hi Asela,
>
> On Fri, Jul 14, 2017 at 9:34 AM, Asela Pathberiya 
> wrote:
>
>>
>>
>> On Fri, Jul 14, 2017 at 11:31 AM, Harsha Kumara 
>> wrote:
>>
>>> Hi All,
>>>
>>> This is regarding the behavior of Authentication flow between
>>> multiple service providers.
>>>
>>> I have created two service providers with following configurations.
>>>
>>> *SP1*
>>>
>>> This service provider has two options which allow to users to login
>>> either Basic Authentication scheme or Facebook
>>>
>>> Configuration
>>>
>>> 1 Authentication Step with MultiOption with Basic Auth and Facebook.
>>>
>>>
>>> *SP2*
>>>
>>> This service provider has two authentication steps which allow to
>>> users to login either Basic Authentication scheme or Facebook and second
>>> authentication step with TOTP.
>>>
>>> Configuration
>>>
>>> 2 Authentication Steps
>>>
>>>- 1 Authentication Step with MultiOption with Basic Auth and
>>>Facebook.
>>>- 2 Authentication Step with TOTP
>>>
>>>
>>> *Behavioral Concern*
>>>
>>> I have configured two applications with SP1 and SP2 respectively.
>>> Then I have logged into the first application with Basic Authentication
>>> Scheme which is configured in SP1.
>>>
>>> But when I going to authentication with my second application which
>>> configured with SP2, I have logged into it automatically.
>>>
>>> Shouldn't it ask for TOTP authentication? Because first application
>>> only authenticated with Basic Auth but my second application required 
>>> Basic
>>> Auth + TOTP.
>>>
>>
>> Yes. It should...  Session contains the authenticated SP
>> details.  Therefore;  it can decide based on the SP...  If it is not
>> working,  it seems like a bug..
>>
>
> I don't think it has ever worked like that because we maintain a list
> of IdPs against which we have authenticated so far for any service
> provider. We don't consider the step number here. So since basic auth and
> TOTP both are for LOCAL IdP, IS considers SP2 also authenticated if we
> authenticate with username/password only for SP1.

Re: [Architecture] IS Authentication Flow Behavior Between Multiple Service Providers

[Johann Nallathamby]

On Sat, Jul 15, 2017 at 5:55 AM Darshana Gunawardana 
wrote:

> Hi Johann,
>
> On Fri, Jul 14, 2017 at 5:45 PM Johann Nallathamby 
> wrote:
>
>> Can we change the implementation as follows:
>>
>> If user is going to login to application Y, which has steps 1 to *m*,
>> and user has already logged into Y
>>
> Should be corrected as application X?
>

Yes :)


> which has steps 1 to *n*,
>>
> given p <= m , n
>> User is already authenticated to step *p* of application Y, if user has
>> logged in from any of the optional IdPs in step *p*, within the sequence
>> of application X, between steps *p* and n*.*
>>
>
> Anyway I don't think above algorithm is correct because it base on number
> of steps.
>
> 1. The simple way of fixing this is handle LOCAL idp specially.
>
> Currently the framework calculates to bypass the step p (when loging to
> app Y), if user have already logged in at least one of optional IdP in
> previous login (app X).
>
> We have to add extra level of validation if the previously authenticated
> IdP picked as the LOCAL IdP where it should check the whether previously
> authenticated authenticator (basic authenticator) and current available
> authenticator (TOTP) same or not.
>
> If it's not same, prompt for authentication without bypassing.
>

+1. This is also a good idea.


> 2. The generic way of fixing this is, consider ACR for each IdP
>
> Extending the previous solution, (rather having special checks for LOCAL
> IdP), check on the authentication level / ACR of the  any IdP when it
> decide whether user already logged in with that IdP.
>

I believe that is the solution coming with IS 5.5.0.

>
> WDYT?
>
> Regards,
>
>
>> Can we do something like this for 5.4.0? IMO this will be more practical
>> and step up authentication will work seamlessly.
>>
>> Regards,
>> Johann.
>>
>> On Fri, Jul 14, 2017 at 10:58 AM, Ishara Karunarathna 
>> wrote:
>>
>>> Hi Johan,
>>>
>>> On Fri, Jul 14, 2017 at 1:56 PM, Johann Nallathamby 
>>> wrote:
>>>
 Hi Asela,

 On Fri, Jul 14, 2017 at 9:34 AM, Asela Pathberiya 
 wrote:

>
>
> On Fri, Jul 14, 2017 at 11:31 AM, Harsha Kumara 
> wrote:
>
>> Hi All,
>>
>> This is regarding the behavior of Authentication flow between
>> multiple service providers.
>>
>> I have created two service providers with following configurations.
>>
>> *SP1*
>>
>> This service provider has two options which allow to users to login
>> either Basic Authentication scheme or Facebook
>>
>> Configuration
>>
>> 1 Authentication Step with MultiOption with Basic Auth and Facebook.
>>
>>
>> *SP2*
>>
>> This service provider has two authentication steps which allow to
>> users to login either Basic Authentication scheme or Facebook and second
>> authentication step with TOTP.
>>
>> Configuration
>>
>> 2 Authentication Steps
>>
>>- 1 Authentication Step with MultiOption with Basic Auth and
>>Facebook.
>>- 2 Authentication Step with TOTP
>>
>>
>> *Behavioral Concern*
>>
>> I have configured two applications with SP1 and SP2 respectively.
>> Then I have logged into the first application with Basic Authentication
>> Scheme which is configured in SP1.
>>
>> But when I going to authentication with my second application which
>> configured with SP2, I have logged into it automatically.
>>
>> Shouldn't it ask for TOTP authentication? Because first application
>> only authenticated with Basic Auth but my second application required 
>> Basic
>> Auth + TOTP.
>>
>
> Yes. It should...  Session contains the authenticated SP details.
> Therefore;  it can decide based on the SP...  If it is not working,  it
> seems like a bug..
>

 I don't think it has ever worked like that because we maintain a list
 of IdPs against which we have authenticated so far for any service
 provider. We don't consider the step number here. So since basic auth and
 TOTP both are for LOCAL IdP, IS considers SP2 also authenticated if we
 authenticate with username/password only for SP1.

 We were actually trying this as a logical workaround to have step up
 authentication so when the application decided it needs 2nd factor
 authentication to send a request using the new service provider ID, but it
 seems not possible with current version of IS.

>>> I tested this and got the same behavior.
>>> As I remember we have authenticator information as well in the context
>>> so we should be able to fix this in future.
>>>
>>> -Ishara
>>>

 Regards,
 Johann.


> Thanks,
> Asela.
>
>
>>
>> Thanks,
>> Harsha
>>
>> --
>> Harsha Kumara
>> Software Engineer, WSO2 Inc.
>> Mobile: