To make things a little bit more specific this is the kind of thing we are
seeing.. I have chopped out some line, etc to make it more readable.
Note that during this one minute interval the read_pos went from
177914 to 178522
which is a difference of 608 yet we processed 4140 tuples based on the
execute-count.
Maybe we are just interpreting these stats incorrectly.....
2017-06-05 13:50:17.834 EMSMetricsConsumer 10:B_CALC :8__receive
{arrival_rate_secs=9.876286516269882, overflow=0, *read_pos=177914*,
write_pos=177915, sojourn_time_ms=101.25263157894737, capacity=16384,
population
2017-06-05 13:51:17.836 EMSMetricsConsumer 10:B_CALC :8 __receive
{arrival_rate_secs=10.809687142708658, overflow=0, *read_pos=178522*,
write_pos=178523, sojourn_time_ms=92.50961538461539, capacity=16384,
population
=1}
2017-06-05 13:51:17.837 EMSMetricsConsumer 10:B_CALC :8 __execute-count {
*B_MARKET:TRADE_STREAM=4140*, B_TIMER:DATE_TICK=0, B_MARKET:OPTX_STREAM=0}
On Mon, Jun 5, 2017 at 9:10 PM, Roshan Naik <[email protected]> wrote:
>
>
>
>
> AM> So the difference in the read and write queue positions will be 100
> then, correct?
>
>
>
> Only if the diff was 0 prior to the publish. More accurately… the diff
> between the read and write positions will increase by 100 after the publish.
>
>
>
>
>
> AM> Empirically what we are seeing would lead us to believe that each
> queue entry is actually a list of tuples, not a single tuple.
>
>
>
> It is not the case with the current code in master/1.x branches. Not sure
> if it used to be different previously.
>
>
>
> AM> There is a pretty old description of this behavior by Michael Noll @
> http://www.michael-noll.com/blog/2013/06/21/understanding-
> storm-internal-message-buffers/. Very good write-up but wondering if
> it's still appliest.
>
>
>
>
>
> Michael’s otherwise excellent description seems a bit dated now. Not very
> sure, but that behavior may have been true when he wrote the blog.
>
>
>
> I give a more detailed and up-to-date diagram/description of current
> messaging system here (starting 36:00)
>
> https://www.youtube.com/watch?v=kCRv6iEd7Ow
>
>
>
> With the diagram show there, here is more info on the write path:
>
>
>
> - The incoming writes are buffered into an ArrayList<Object>
> called ‘currentBatch’
>
> - If currenBatch is full then we try to drain it into the
> Disruptor (as I described previously… one element at a time.. followed by a
> single publish).
>
> - But if Disruptor is full, the entire currentBatch list is
> inserted as **one element** into another (unbounded) overflow list which
> is of type ConcurrentLinkedQueue<ArrayList<Object>>. The currentBatch is
> then cleared to make room for new incoming events
>
> - Every once in a while a flusher thread comes along and tries
> to drain any available items in overflow list into the Disruptor.
>
>
>
> Having said that, we are planning to significantly revise this messaging
> subsystem for 2.0 as explained later in the above video.
>
>
>
> -roshan
>
>
>
>
>
