date:20170227

Re: [Architecture] [IoT] Load Distribution for Device + Server Communication.

2017-02-27 Thread Menaka Jayawardena

Hi,

IMO we can send a timeout with the operation which, once the operation/
notification is received by the device, it waits for the given timeout and
respond back to the server to perform the operation. All operations are
sent to devices at ones and the timeout should be different to each device.
So that each device gets its own time to contact the server and complete
the operation.

Also, this can be used for batch processing as well.
As an example, if there are 1000 devices, for first 100 devices, the
timeout will be 0s, for next 100, 10s and so on. Then the first batch will
come to the server just after they received the operation and the next
batch will wait for 10s(or any given time) to respond back. This will
reduce the overhead of waiting for a set of devices to complete the
operation.

I think through this we can limit the number of devices which contact the
server at a given time and handle a reasonable amount.

Appreciate any feedback.

Thanks and regards,
Menaka


On Mon, Feb 27, 2017 at 11:09 PM, Geeth Munasinghe  wrote:

> Hi all,
>
> We have encountered an issue with balancing load distribution of device +
> server communication. Device and server communication happens through
> following ways.
>
>1. Polling - Device keeps polling the server
>2. Push notification - Server sends a notification to device to come
>retrieve the operation. (wake up call)
>3. Push message - Whole message is pushed to device as notification.
>
> In 1 and 2 ways, device always comes to server to pick up the operation.
> (In MDM scenarios, we use only 1 and 2) We have found few scenarios where
> this would lead to a request overload to the server.
>
>1. Device information retrieval
>2. Device application list retrieval
>3. Device location retrieval
>4. Policy monitoring
>5. Installing/Uninstalling application (Role based)
>
> First four scenarios are handled by background tasks. These tasks are
> responsible for adding an operation against each of the device. This
> process would notify the device regarding operation or device will come to
> server at a pre-configured time to retrieve the operation. Specially with
> the notification process, when an operation is added to all the device,
> notification is send to all of them. In iOS, we send to the notification to
> APNS, for android - GCM/FCM and for Windows - WNS. These notification
> services normally deliver the messages to devices quickly. Therefore as
> soon as device receives the notification, it starts to contact the server.
> If this happens with a server which has 1,000s of devices enrolled, most of
> devices start retrieving the operation from the server which leads to a
> request a burst. 5th scenario happens when we install an application to a
> role which has 1,000s of users. This also starts sending notifications to
> all the devices of those users, which would result in a request burst.
>
> As a solution, we can limit the number of devices notified at given
> moment. Only set of devices will receive the notification at first, then
> after some time interval second set of devices will receive the
> notification and continue until all the devices are notified. This way, we
> can make sure only chunk of devices receives the notification, hence reduce
> the server load.
>
> Please give your feedbacks and suggestions.
>
> Thanks
> Geeth
>
> --
>
> *G. K. S. Munasinghe*
> *Senior Software Engineer,*
> *WSO2, Inc. http://wso2.com  *
> *lean.enterprise.middleware.*
>
> email: ge...@wso2.com
> phone:(+94) 777911226 <+94%2077%20791%201226>
>
> 
>



-- 
*Menaka Jayawardena*
*Software Engineer - WSO2 Inc*
*Tel : 071 350 5470*
*LinkedIn: https://lk.linkedin.com/in/menakajayawardena
*
*Blog: https://menakamadushanka.wordpress.com/
*
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [IoT] Load Distribution for Device + Server Communication.

2017-02-27 Thread Charitha Goonetilleke

Hi All,

I think it is better to have a solution which could be implemented in
server side to distribute operations among devices. Because as we know
device agents in IoTS devices are different to each other. In according to
our architecture, we are letting anyone to write their own device types as
well as the agents. Also we don't have our own agent for windows devices
but using default windows workspace configurations to provide MDM. So
implementing agent side solution would be more complex for some device
types and will not move forward with all device types.

Anyway as according what to suggested by Geeth & Menaka, we could use batch
processing in server side to deliver operation to device batches rather
than sending it to all devices. So we can use operation queue to deliver
operations periodically to device batches. However we need to figure out
few more things in there.

   1. How we figure out optimum number of devices we should call in a batch?
   2. What is the time span of delivering operation to all target devices?
   3. What we do if someone want to send operation immediately?
   4. Can we assure guaranteed delivery and can avoid repeated operations?

For 1 & 2, we could use configuration to provide capability to set size of
a batch and time span. However for 3, we need to have additional option to
send operations immediately, probably using a priority queue. Also when it
comes to 4, we need to keep records about the operations we have delivered
and would be enough to add one additional operation status like "Queued".
So we can change operation status to pending, when operation is delivered.

WDYT?

Thanks & Regards,
/charithag

On Tue, Feb 28, 2017 at 12:39 AM, Menaka Jayawardena 
wrote:

> Hi,
>
> IMO we can send a timeout with the operation which, once the operation/
> notification is received by the device, it waits for the given timeout and
> respond back to the server to perform the operation. All operations are
> sent to devices at ones and the timeout should be different to each device.
> So that each device gets its own time to contact the server and complete
> the operation.
>
> Also, this can be used for batch processing as well.
> As an example, if there are 1000 devices, for first 100 devices, the
> timeout will be 0s, for next 100, 10s and so on. Then the first batch will
> come to the server just after they received the operation and the next
> batch will wait for 10s(or any given time) to respond back. This will
> reduce the overhead of waiting for a set of devices to complete the
> operation.
>
> I think through this we can limit the number of devices which contact the
> server at a given time and handle a reasonable amount.
>
> Appreciate any feedback.
>
> Thanks and regards,
> Menaka
>
>
> On Mon, Feb 27, 2017 at 11:09 PM, Geeth Munasinghe  wrote:
>
>> Hi all,
>>
>> We have encountered an issue with balancing load distribution of device +
>> server communication. Device and server communication happens through
>> following ways.
>>
>>1. Polling - Device keeps polling the server
>>2. Push notification - Server sends a notification to device to come
>>retrieve the operation. (wake up call)
>>3. Push message - Whole message is pushed to device as notification.
>>
>> In 1 and 2 ways, device always comes to server to pick up the operation.
>> (In MDM scenarios, we use only 1 and 2) We have found few scenarios where
>> this would lead to a request overload to the server.
>>
>>1. Device information retrieval
>>2. Device application list retrieval
>>3. Device location retrieval
>>4. Policy monitoring
>>5. Installing/Uninstalling application (Role based)
>>
>> First four scenarios are handled by background tasks. These tasks are
>> responsible for adding an operation against each of the device. This
>> process would notify the device regarding operation or device will come to
>> server at a pre-configured time to retrieve the operation. Specially with
>> the notification process, when an operation is added to all the device,
>> notification is send to all of them. In iOS, we send to the notification to
>> APNS, for android - GCM/FCM and for Windows - WNS. These notification
>> services normally deliver the messages to devices quickly. Therefore as
>> soon as device receives the notification, it starts to contact the server.
>> If this happens with a server which has 1,000s of devices enrolled, most of
>> devices start retrieving the operation from the server which leads to a
>> request a burst. 5th scenario happens when we install an application to a
>> role which has 1,000s of users. This also starts sending notifications to
>> all the devices of those users, which would result in a request burst.
>>
>> As a solution, we can limit the number of devices notified at given
>> moment. Only set of devices will receive the notification at first, then
>> after some time interval second set of devices will receive the
>> notification and continue until a

Re: [Architecture] IdentityStore APIs in C5

2017-02-27 Thread Gayan Gunawardana

On Mon, Feb 27, 2017 at 6:52 PM, Omindu Rathnaweera  wrote:

> We already have a JIRA[1] and a redmine for this.
>
> One requirement I came across was to identify when we are adding a
> duplicate user. ATM we don't have a method to identify without checking
> 'isUserExist'. It's better if we can at least introduce an error code for
> this.
>
> @Gayan might have come across similar requirements while developing the
> SCIM component.
>
Yes.

>
> [1] - https://wso2.org/jira/browse/IDENTITY-5768
>
> On Mon, Feb 27, 2017 at 6:31 PM, Thanuja Jayasinghe 
> wrote:
>
>> Hi Gayan,
>>
>> We have already defined an exception hierarchy for identity components.
>>
>> IdentityException[1]
>> ├── IdentityServerException[2]
>> └── IdentityClientException[3]
>>
>>  All exceptions classes defined for identity components extend either 
>> IdentityServerException
>> or IdentityClientException.
>>
>> So any client can catch the exception as follows,
>>
>> catch (IdentityClientException ex) {
>> // Can return the same error message and code to the client
>> } catch (IdentityException ex) {
>> // Need to log and return a generic message to the client
>> }
>>
> This is fine when we propagate exception to out side. IdentityStore APIs
are extensively used by other identity components where they want to take
decisions based on exceptions provided by IdentityStore APIs. Just by
having IdentityServerException, IdentityClientException does not provide
exact idea to API consumer.

I guess answer to above question is having error codes. As I mentioned in
the initial mail, isn't it better to have exception hierarchy than error
codes ?


>> Since we add these exception classes recently, we need to
>> update carbon-identity-mgt repo.
>>
>> [1] - https://github.com/wso2/carbon-identity-commons/blob/maste
>> r/components/org.wso2.carbon.identity.common/src/main/java/
>> org/wso2/carbon/identity/common/base/exception/IdentityException.java
>> [2] - https://github.com/wso2/carbon-identity-commons/blob/maste
>> r/components/org.wso2.carbon.identity.common/src/main/java/
>> org/wso2/carbon/identity/common/base/exception/Identity
>> ServerException.java
>> [3] - https://github.com/wso2/carbon-identity-commons/blob/maste
>> r/components/org.wso2.carbon.identity.common/src/main/java/
>> org/wso2/carbon/identity/common/base/exception/Identity
>> ClientException.java
>>
>> Thanks,
>> Thanuja
>>
>> On Mon, Feb 27, 2017 at 10:06 AM, Ruwan Abeykoon  wrote:
>>
>>> Hi All,
>>> +1 to have an exception hierarchy, which carries information for
>>> specific errors.
>>>
>>> I think we should follow the way Java IO exceptions are done.
>>>
>>> Cheers,
>>> Ruwan
>>>
>>>
>>> On Mon, Feb 27, 2017 at 9:58 AM, Gayan Gunawardana 
>>> wrote:
>>>
 Hi All,

 Shall we revisit IdentityStore APIs? For an example addUser method[1]
 throws IdentityStoreClientException and IdentityStoreServerException
 in many cases where client cannot differentiate the reason. There will be
 relevant error message but client cannot rely on error message to take
 decisions.
 IMO we should have proper exception hierarchy or error codes. I'm +1 to
 have
 exception hierarchy.

 WDYT ?

 [1] https://github.com/wso2/carbon-identity-mgt/blob/master/comp
 onents/org.wso2.carbon.identity.mgt/src/main/java/org/wso2/c
 arbon/identity/mgt/impl/IdentityStoreImpl.java#L985

 Thanks,
 Gayan
 --
 Gayan Gunawardana
 Software Engineer; WSO2 Inc.; http://wso2.com/
 Email: ga...@wso2.com
 Mobile: +94 (71) 8020933

>>>
>>>
>>>
>>> --
>>>
>>> *Ruwan Abeykoon*
>>> *Associate Director/Architect**,*
>>> *WSO2, Inc. http://wso2.com  *
>>> *lean.enterprise.middleware.*
>>>
>>>
>>
>>
>> --
>> *Thanuja Lakmal*
>> Senior Software Engineer
>> WSO2 Inc. http://wso2.com/
>> *lean.enterprise.middleware*
>> Mobile: +94715979891 +94758009992
>>
>
>
>
> --
> Omindu Rathnaweera
> Software Engineer, WSO2 Inc.
> Mobile: +94 771 197 211 <+94%2077%20119%207211>
>



-- 
Gayan Gunawardana
Software Engineer; WSO2 Inc.; http://wso2.com/
Email: ga...@wso2.com
Mobile: +94 (71) 8020933
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

[Architecture] [IoT] Load Distribution for Device + Server Communication.

2017-02-27 Thread Geeth Munasinghe

Hi all,

We have encountered an issue with balancing load distribution of device +
server communication. Device and server communication happens through
following ways.

   1. Polling - Device keeps polling the server
   2. Push notification - Server sends a notification to device to come
   retrieve the operation. (wake up call)
   3. Push message - Whole message is pushed to device as notification.

In 1 and 2 ways, device always comes to server to pick up the operation.
(In MDM scenarios, we use only 1 and 2) We have found few scenarios where
this would lead to a request overload to the server.

   1. Device information retrieval
   2. Device application list retrieval
   3. Device location retrieval
   4. Policy monitoring
   5. Installing/Uninstalling application (Role based)

First four scenarios are handled by background tasks. These tasks are
responsible for adding an operation against each of the device. This
process would notify the device regarding operation or device will come to
server at a pre-configured time to retrieve the operation. Specially with
the notification process, when an operation is added to all the device,
notification is send to all of them. In iOS, we send to the notification to
APNS, for android - GCM/FCM and for Windows - WNS. These notification
services normally deliver the messages to devices quickly. Therefore as
soon as device receives the notification, it starts to contact the server.
If this happens with a server which has 1,000s of devices enrolled, most of
devices start retrieving the operation from the server which leads to a
request a burst. 5th scenario happens when we install an application to a
role which has 1,000s of users. This also starts sending notifications to
all the devices of those users, which would result in a request burst.

As a solution, we can limit the number of devices notified at given moment.
Only set of devices will receive the notification at first, then after some
time interval second set of devices will receive the notification and
continue until all the devices are notified. This way, we can make sure
only chunk of devices receives the notification, hence reduce the server
load.

Please give your feedbacks and suggestions.

Thanks
Geeth

-- 

*G. K. S. Munasinghe*
*Senior Software Engineer,*
*WSO2, Inc. http://wso2.com  *
*lean.enterprise.middleware.*

email: ge...@wso2.com
phone:(+94) 777911226 <+94%2077%20791%201226>


___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] IdentityStore APIs in C5

2017-02-27 Thread Omindu Rathnaweera

We already have a JIRA[1] and a redmine for this.

One requirement I came across was to identify when we are adding a
duplicate user. ATM we don't have a method to identify without checking
'isUserExist'. It's better if we can at least introduce an error code for
this.

@Gayan might have come across similar requirements while developing the
SCIM component.

[1] - https://wso2.org/jira/browse/IDENTITY-5768

On Mon, Feb 27, 2017 at 6:31 PM, Thanuja Jayasinghe 
wrote:

> Hi Gayan,
>
> We have already defined an exception hierarchy for identity components.
>
> IdentityException[1]
> ├── IdentityServerException[2]
> └── IdentityClientException[3]
>
>  All exceptions classes defined for identity components extend either 
> IdentityServerException
> or IdentityClientException.
>
> So any client can catch the exception as follows,
>
> catch (IdentityClientException ex) {
> // Can return the same error message and code to the client
> } catch (IdentityException ex) {
> // Need to log and return a generic message to the client
> }
>
> Since we add these exception classes recently, we need to
> update carbon-identity-mgt repo.
>
> [1] - https://github.com/wso2/carbon-identity-commons/blob/
> master/components/org.wso2.carbon.identity.common/src/
> main/java/org/wso2/carbon/identity/common/base/
> exception/IdentityException.java
> [2] - https://github.com/wso2/carbon-identity-commons/blob/
> master/components/org.wso2.carbon.identity.common/src/
> main/java/org/wso2/carbon/identity/common/base/exception/
> IdentityServerException.java
> [3] - https://github.com/wso2/carbon-identity-commons/blob/
> master/components/org.wso2.carbon.identity.common/src/
> main/java/org/wso2/carbon/identity/common/base/exception/
> IdentityClientException.java
>
> Thanks,
> Thanuja
>
> On Mon, Feb 27, 2017 at 10:06 AM, Ruwan Abeykoon  wrote:
>
>> Hi All,
>> +1 to have an exception hierarchy, which carries information for specific
>> errors.
>>
>> I think we should follow the way Java IO exceptions are done.
>>
>> Cheers,
>> Ruwan
>>
>>
>> On Mon, Feb 27, 2017 at 9:58 AM, Gayan Gunawardana 
>> wrote:
>>
>>> Hi All,
>>>
>>> Shall we revisit IdentityStore APIs? For an example addUser method[1]
>>> throws IdentityStoreClientException and IdentityStoreServerException in
>>> many cases where client cannot differentiate the reason. There will be
>>> relevant error message but client cannot rely on error message to take
>>> decisions.
>>> IMO we should have proper exception hierarchy or error codes. I'm +1 to
>>> have
>>> exception hierarchy.
>>>
>>> WDYT ?
>>>
>>> [1] https://github.com/wso2/carbon-identity-mgt/blob/master/comp
>>> onents/org.wso2.carbon.identity.mgt/src/main/java/org/wso2/
>>> carbon/identity/mgt/impl/IdentityStoreImpl.java#L985
>>>
>>> Thanks,
>>> Gayan
>>> --
>>> Gayan Gunawardana
>>> Software Engineer; WSO2 Inc.; http://wso2.com/
>>> Email: ga...@wso2.com
>>> Mobile: +94 (71) 8020933
>>>
>>
>>
>>
>> --
>>
>> *Ruwan Abeykoon*
>> *Associate Director/Architect**,*
>> *WSO2, Inc. http://wso2.com  *
>> *lean.enterprise.middleware.*
>>
>>
>
>
> --
> *Thanuja Lakmal*
> Senior Software Engineer
> WSO2 Inc. http://wso2.com/
> *lean.enterprise.middleware*
> Mobile: +94715979891 +94758009992
>



-- 
Omindu Rathnaweera
Software Engineer, WSO2 Inc.
Mobile: +94 771 197 211
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] IdentityStore APIs in C5

2017-02-27 Thread Thanuja Jayasinghe

Hi Gayan,

We have already defined an exception hierarchy for identity components.

IdentityException[1]
├── IdentityServerException[2]
└── IdentityClientException[3]

 All exceptions classes defined for identity components extend either
IdentityServerException
or IdentityClientException.

So any client can catch the exception as follows,

catch (IdentityClientException ex) {
// Can return the same error message and code to the client
} catch (IdentityException ex) {
// Need to log and return a generic message to the client
}

Since we add these exception classes recently, we need to
update carbon-identity-mgt repo.

[1] -
https://github.com/wso2/carbon-identity-commons/blob/master/components/org.wso2.carbon.identity.common/src/main/java/org/wso2/carbon/identity/common/base/exception/IdentityException.java
[2] -
https://github.com/wso2/carbon-identity-commons/blob/master/components/org.wso2.carbon.identity.common/src/main/java/org/wso2/carbon/identity/common/base/exception/IdentityServerException.java
[3] -
https://github.com/wso2/carbon-identity-commons/blob/master/components/org.wso2.carbon.identity.common/src/main/java/org/wso2/carbon/identity/common/base/exception/IdentityClientException.java

Thanks,
Thanuja

On Mon, Feb 27, 2017 at 10:06 AM, Ruwan Abeykoon  wrote:

> Hi All,
> +1 to have an exception hierarchy, which carries information for specific
> errors.
>
> I think we should follow the way Java IO exceptions are done.
>
> Cheers,
> Ruwan
>
>
> On Mon, Feb 27, 2017 at 9:58 AM, Gayan Gunawardana  wrote:
>
>> Hi All,
>>
>> Shall we revisit IdentityStore APIs? For an example addUser method[1]
>> throws IdentityStoreClientException and IdentityStoreServerException in
>> many cases where client cannot differentiate the reason. There will be
>> relevant error message but client cannot rely on error message to take
>> decisions.
>> IMO we should have proper exception hierarchy or error codes. I'm +1 to
>> have
>> exception hierarchy.
>>
>> WDYT ?
>>
>> [1] https://github.com/wso2/carbon-identity-mgt/blob/master/
>> components/org.wso2.carbon.identity.mgt/src/main/java/
>> org/wso2/carbon/identity/mgt/impl/IdentityStoreImpl.java#L985
>>
>> Thanks,
>> Gayan
>> --
>> Gayan Gunawardana
>> Software Engineer; WSO2 Inc.; http://wso2.com/
>> Email: ga...@wso2.com
>> Mobile: +94 (71) 8020933
>>
>
>
>
> --
>
> *Ruwan Abeykoon*
> *Associate Director/Architect**,*
> *WSO2, Inc. http://wso2.com  *
> *lean.enterprise.middleware.*
>
>


-- 
*Thanuja Lakmal*
Senior Software Engineer
WSO2 Inc. http://wso2.com/
*lean.enterprise.middleware*
Mobile: +94715979891 +94758009992
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [IoT] Load Distribution for Device + Server Communication.

Re: [Architecture] [IoT] Load Distribution for Device + Server Communication.

Re: [Architecture] IdentityStore APIs in C5

[Architecture] [IoT] Load Distribution for Device + Server Communication.

Re: [Architecture] IdentityStore APIs in C5

Re: [Architecture] IdentityStore APIs in C5

6 matches

Site Navigation

Mail list logo

Footer information