AM> So the difference in the read and write queue positions will be 100 then, correct?
Only if the diff was 0 prior to the publish. More accurately… the diff between the read and write positions will increase by 100 after the publish. AM> Empirically what we are seeing would lead us to believe that each queue entry is actually a list of tuples, not a single tuple. It is not the case with the current code in master/1.x branches. Not sure if it used to be different previously. AM> There is a pretty old description of this behavior by Michael Noll @http://www.michael-noll.com/blog/2013/06/21/understanding-storm-internal-message-buffers/. Very good write-up but wondering if it's still appliest. Michael’s otherwise excellent description seems a bit dated now. Not very sure, but that behavior may have been true when he wrote the blog. I give a more detailed and up-to-date diagram/description of current messaging system here (starting 36:00) https://www.youtube.com/watch?v=kCRv6iEd7Ow With the diagram show there, here is more info on the write path: - The incoming writes are buffered into an ArrayList<Object> called ‘currentBatch’ - If currenBatch is full then we try to drain it into the Disruptor (as I described previously… one element at a time.. followed by a single publish). - But if Disruptor is full, the entire currentBatch list is inserted as *one element* into another (unbounded) overflow list which is of type ConcurrentLinkedQueue<ArrayList<Object>>. The currentBatch is then cleared to make room for new incoming events - Every once in a while a flusher thread comes along and tries to drain any available items in overflow list into the Disruptor. Having said that, we are planning to significantly revise this messaging subsystem for 2.0 as explained later in the above video. -roshan
