Currently we are using 1.0.1. Thanks for the video. It definitely clarifies the lower level details of the queueing.
Quick followup question. In the case where the disruptor queue is full and data ends up in the overflow list, you mentioned that **one element** in that list can be multiple tuples.. When the flusher eventually drains items from the overflow list, do they end up as individual entries in the disruptor queue? For example if 100 tuples ended up in the overflow queue as one element, would the flusher thread eventually insert one element composed of the 100 tuples in to the disruptor queue or 100 elements? Thanks! On Mon, Jun 5, 2017 at 9:10 PM, Roshan Naik <[email protected]> wrote: > > > > > AM> So the difference in the read and write queue positions will be 100 > then, correct? > > > > Only if the diff was 0 prior to the publish. More accurately… the diff > between the read and write positions will increase by 100 after the publish. > > > > > > AM> Empirically what we are seeing would lead us to believe that each > queue entry is actually a list of tuples, not a single tuple. > > > > It is not the case with the current code in master/1.x branches. Not sure > if it used to be different previously. > > > > AM> There is a pretty old description of this behavior by Michael Noll @ > http://www.michael-noll.com/blog/2013/06/21/understanding- > storm-internal-message-buffers/. Very good write-up but wondering if > it's still appliest. > > > > > > Michael’s otherwise excellent description seems a bit dated now. Not very > sure, but that behavior may have been true when he wrote the blog. > > > > I give a more detailed and up-to-date diagram/description of current > messaging system here (starting 36:00) > > https://www.youtube.com/watch?v=kCRv6iEd7Ow > > > > With the diagram show there, here is more info on the write path: > > > > - The incoming writes are buffered into an ArrayList<Object> > called ‘currentBatch’ > > - If currenBatch is full then we try to drain it into the > Disruptor (as I described previously… one element at a time.. followed by a > single publish). > > - But if Disruptor is full, the entire currentBatch list is > inserted as **one element** into another (unbounded) overflow list which > is of type ConcurrentLinkedQueue<ArrayList<Object>>. The currentBatch is > then cleared to make room for new incoming events > > - Every once in a while a flusher thread comes along and tries > to drain any available items in overflow list into the Disruptor. > > > > Having said that, we are planning to significantly revise this messaging > subsystem for 2.0 as explained later in the above video. > > > > -roshan > > > > >
