Out of order processing always implies a waiting time and timeout events
processing. What is hard, is discovering how long to wait. What hasn’t been
happening by and large, is to have self adjusting wait times that will allow
network latency to be considered and handled effectively. Additionally, there
is the whole issue of UDP (unacknowledged, non-streaming data flow) verses TCP
(streaming, guaranteed ordering) as a transport.
There are several places in this time I’m my career where I’ve actually needed
and wanted to write a PID based networking retry and flow management library.
The idea being that one can learn about intervals of time that typically are in
processing loops across networking stacks and adjust software behaviors to deal
with anomalous behaviors and optimize for patterns of success or failure.
TCP’s
Syn->
<-SynACK
SynACK->
Pattern is an algorithm which attempts to make sure that traffic is flowing in
both directions and that both ends of a TCP socket connection agree on the
state of the socket. However, the synack timeout on the connection initiator
causes “resources” to be reserved for a period of time. In the beginning, the
TCP connection timeout was on the order of minutes, because “modems” had to
dial the connection in many cases. In this day and age, the timeout can be
more practically set to be on the order of a few seconds because of the speed
of networks and the “always connected” state.
But, imagine what could happen if the timeout is set too small. Suddenly,
networking stops functioning to certain places where the timeout is not long
enough. This timeout could be adjusted in length by first noting how fast
connections occur. It could be lengthened if the SynACK comes back after the
timeout, but that could only happen if we still retained knowledge of the last
attempted connections that failed due to timeout, or else malicious SynACK
attacks could create problems.
So, typically, behavior based tuning of networking algorithms has not been done
because there is always a “hard” case that seems to create enough complication
to keep things from being “helpful” overall.
I think that we really need to think more about how to make notifications more
stateless so that there is an opportunity to not have to consider history.
Gregg Wonderly
On Apr 19, 2014, at 4:44 AM, Peter <j...@zeus.net.au> wrote:
> Thanks Gregg,
>
> For now, with SDM I'm using a PriorityBlockingQueue for buffering events.
> Incoming threads originating from remote calls, place an event into the
> buffer, sleep briefly, then take an event from the buffer and continue their
> execution path through SDM. I did consider using a handoff thread, but
> there's no contention in any tests, so I kept it simple. While in the
> buffer, the events are re ordered by the Comparator.
>
> When SDM receives events out of order it performs a lookup network call,
> reordering helps to avoid that.
>
> Refactoring SDM to fix race conditions and sync bugs has taken some months,
> but it's sorted now.
>
> The latest issue now appears to be related to building on Jenkins, creating
> jar files and Processes using AppClassLoader resulting in
> ClassNotFoundExceptions.
>
> The cause of CNFE dosn't appear related to River code, but I'm investigating
> whether there are any issues with the build process, it doesn't appear to be
> a classdep or asm problem, as all not found classes are accounted for, it
> could be a classpath issue.
>
> If you profile Jini / River code now, the hotspots are all java sockets, no
> River code is a hotspot, ClassLoading is magnitudes of order faster and there
> is no contention whatsoever.
>
> Cheers,
>
> Peter.
>
> ----- Original message -----
>> The simple programming mechanism I use for unordered but inclusive
>> events is a map or set. I use a map for what is expected and a map for
>> what has happened. I use a thread responding to notifications to fill
>> in the results data and then either call out or notify another thread of
>> the new results. It’s that final code activity that checks for “do I
>> have everything I need to do more work?” It will then react when that
>> moment occurs, retry, re-dispatch or whatever the appropriate action is.
>> That way, everything is separated and still involves testable behaviors.
>>
>> Gregg Wonderly
>>
>> On Apr 17, 2014, at 7:56 PM, Peter <j...@zeus.net.au> wrote:
>>
>>> Thanks Greg, I agree, remote events have an event id and sequence
>>> number, so it's very easy for clients to order them if necessary.
>>>
>>> I think for the test I'll create a simple comparator that orders the
>>> events at the client.
>>>
>>> The test only needs to ensure that all expected events are received
>>> and provide sufficient information allowing them to be correctly
>>> ordered.
>>>
>>> Regards,
>>>
>>> Peter.
>>>
>>> ----- Original message -----
>>>>
>>>> Hi Peter:
>>>>
>>>> You should probably create a JIRA enhancement ticket to track
>>>> discussion if you’re picturing adding some kind of
>>>> order-guaranteeing comparator to the API. But I don’t think you
>>>> really need to do that, because the usage would be so dependent on
>>>> the client’s architecture that it probably isn’t sensible to put it
>>>> in the API.
>>>>
>>>> On the actual question, I’d suggest that Reggie should make no
>>>> guarantees on the order of event delivery (as per the event spec).
>>>> That being the case, imposing some kind of order is a client
>>>> problem, not Reggie’s. I would suggest modifying the test simply
>>>> to ensure that all the expected events have been received in the
>>>> required time, regardless of the order. Perhaps also add some
>>>> clarification to the service registrar spec.
>>>>
>>>> Cheers,
>>>>
>>>> Greg Trasuk.
>>>>
>>>> On Apr 17, 2014, at 7:30 AM, Peter <j...@zeus.net.au> wrote:
>>>>
>>>>>
>>>>>
>>>>> From: Peter <j...@zeus.net.au>
>>>>> Subject: RemoteEvent specification - proposal
>>>>> Date: April 17, 2014 at 7:28:13 AM EDT
>>>>> To: d...@apache.river.org
>>>>> Reply-To: Peter <j...@zeus.net.au>
>>>>>
>>>>>
>>>>> The Jini Remote Event specification clearly states that remote
>>>>> events may arrive out of order, yet some lookup tck tests in the
>>>>> qa test suite require events to arrive in order.
>>>>>
>>>>> Presently I have an Executor in Reggie, used specifically for
>>>>> sending event notifications, however it is single threaded, to
>>>>> ensure events arrive in an order identical to client registration,
>>>>> to avoid qa test failures.
>>>>>
>>>>> I propose creating a comparator clients can use to order events as
>>>>> they arrive. This will allow qa tests, when utilising this
>>>>> comparator, to pass when Reggie is configured to use a multi
>>>>> threaded event notifier executor. This would increase Reggie's
>>>>> scalability for event notifications.
>>>>>
>>>>> Thoughts?
>>>>>
>>>>> Regards,
>>>>>
>>>>> Peter.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
