Had an offline chat with Suho and found out that following are the reasons
for having a receiver queue;
* In HA mode there's a "state" sync happening at the startup of a DAS node.
We need to buffer the events that we may receive during this time.
* Currently, the "event" sync is a synchronous operation; hence we need to
avoid back-pressuring the client.
* There's a periodic state sync which is happening and the receivers will
be paused during that time, hence we need to buffer the events.
On Wed, Jan 31, 2018 at 6:05 PM, Sriskandarajah Suhothayan <s...@wso2.com>
wrote:
>
> On Wed, Jan 31, 2018 at 4:56 PM Nirmal Fernando <nir...@wso2.com> wrote:
>
>> Suho,
>>
>> Thanks for the reply. What I am thinking is why we need a receiver queue,
>> if we made DAS1->DAS2 async?
>>
> How will this fix the issue?
>
>>
>>
>> On Wednesday, January 31, 2018, Sriskandarajah Suhothayan <s...@wso2.com>
>> wrote:
>>
>>> AFAIR the event queue is used for buffering the events when states are
>>> being synced, else there will be back pressure on the clients.
>>> I think the rationale for using the same thread with synchronous call
>>> and performing operations in both DAS1 and DAS2 ( in the order DAS1 -> DAS2
>>> -> DAS2-Siddhi -> DAS1-Siddhi) is to make sure both nodes have processed
>>> the same events at the same time.
>>>
>>> I think event processing can be made asynchronous, such that the DAS1's
>>> threads send the event to DAS2 and then returns (DAS1 -> DAS2 -> DAS1), and
>>> a separate thread pool can handle sending events into DAS2-Siddhi.
>>> One possible option is depicted in the image attached.
>>>
>>> WDYT?
>>>
>>> On Wed, Jan 31, 2018 at 1:15 PM, Nirmal Fernando <nir...@wso2.com>
>>> wrote:
>>>
>>>> Thanks, Grainier. Please see my comments inline.
>>>>
>>>> On Wed, Jan 31, 2018 at 1:07 PM, Grainier Perera <grain...@wso2.com>
>>>> wrote:
>>>>
>>>>> InputEventDispatcher will send the event to the callback immediately
>>>>> while QueueInputEventDispatcher will queue events first and there will be
>>>>> an internal worker (QueueInputEventDispatcherWorker) which send the
>>>>> events to the callback.
>>>>>
>>>> The reason to accumulate events in a queue in HA is to be used for
>>>>> event syncing. If the event duplicated in the cluster is set to false,
>>>>> then
>>>>> this queue will be used for the event sync among other nodes in the HA
>>>>> deployment.
>>>>>
>>>>
>>>> Actually it's the same QueueInputEventDispatcherWorker which will do
>>>> the event sync too, AFAIU. But why a queue? Is it because event sync is a
>>>> synchronous operation (DAS1 -> DAS2 -> DAS2-Siddhi -> DAS1)? If so, then my
>>>> next question is, why does the event sync has to be synchronous?
>>>>
>>>>
>>>>> Regards,
>>>>> Grainier
>>>>>
>>>>> On Wed, Jan 31, 2018 at 6:31 AM, Nirmal Fernando <nir...@wso2.com>
>>>>> wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> Can any of you remember the reason for using a receiver queue in HA
>>>>>> mode?
>>>>>>
>>>>>> if (mode == Mode.HA) {
>>>>>> HAConfiguration haConfiguration =
>>>>>> EventReceiverServiceValueHolder.getEventManagementService()
>>>>>> .getManagementModeInfo().getHaConfiguration();
>>>>>> Lock readLock =
>>>>>> EventReceiverServiceValueHolder.getCarbonEventReceiverManagementService().getReadLock();
>>>>>> inputEventDispatcher = new QueueInputEventDispatcher(tenantId,
>>>>>> EventManagementUtil.constructEventSyncId(tenantId,
>>>>>> eventReceiverConfiguration.getEventReceiverName(),
>>>>>> Manager.ManagerType.Receiver),
>>>>>> readLock, exportedStreamDefinition,
>>>>>> haConfiguration.getEventSyncReceiverMaxQueueSizeInMb(),
>>>>>> haConfiguration.getEventSyncReceiverQueueSize());
>>>>>> inputEventDispatcher.setSendToOther(!isEventDuplicatedInCluster);
>>>>>> EventReceiverServiceValueHolder.getEventManagementService()
>>>>>> .registerEventSync((EventSync) inputEventDispatcher,
>>>>>> Manager.ManagerType.Receiver);
>>>>>> } else {
>>>>>> inputEventDispatcher = new InputEventDispatcher();
>>>>>> }
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Thanks & regards,
>>>>>> Nirmal
>>>>>>
>>>>>> Technical Lead, WSO2 Inc.
>>>>>> Mobile: +94715779733 <+94%2071%20577%209733>
>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Grainier Perera
>>>>> Senior Software Engineer
>>>>> Mobile : +94716122384 <+94%2071%20612%202384>
>>>>> WSO2 Inc. | http://wso2.com
>>>>> lean.enterprise.middleware
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Thanks & regards,
>>>> Nirmal
>>>>
>>>> Technical Lead, WSO2 Inc.
>>>> Mobile: +94715779733 <071%20577%209733>
>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> *S. Suhothayan*
>>> Director
>>> *WSO2 Inc. *
>>> http://wso2.com <http://wso2.com/>
>>>
>>>
>>> *cell: (+94) 779 756 757 <+94%2077%20975%206757> | blog:
>>> http://suhothayan.blogspot.com/ <http://suhothayan.blogspot.com/>twitter:
>>> http://twitter.com/suhothayan <http://twitter.com/suhothayan> | linked-in:
>>> http://lk.linkedin.com/in/suhothayan <http://lk.linkedin.com/in/suhothayan>*
>>>
>>
>>
>> --
>>
>> Thanks & regards,
>> Nirmal
>>
>> Technical Lead, WSO2 Inc.
>> Mobile: +94715779733 <+94%2071%20577%209733>
>> Blog: http://nirmalfdo.blogspot.com/
>>
>>
>>
>> --
>
> *S. Suhothayan*
> Director
> *WSO2 Inc. *
> http://wso2.com <http://wso2.com/>
>
>
> *cell: (+94) 779 756 757 <+94%2077%20975%206757> | blog:
> http://suhothayan.blogspot.com/ <http://suhothayan.blogspot.com/>twitter:
> http://twitter.com/suhothayan <http://twitter.com/suhothayan> | linked-in:
> http://lk.linkedin.com/in/suhothayan <http://lk.linkedin.com/in/suhothayan>*
>
--
Thanks & regards,
Nirmal
Technical Lead, WSO2 Inc.
Mobile: +94715779733
Blog: http://nirmalfdo.blogspot.com/
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
