Ok. We do lock all our calls to Basic.sendBinary(), also it seems like moving to Tomcat 8.5 fixes the issue. No proof yet why. Since it always happens on our last write out to a client which should trigger a client ack, the client will immediately send an ack back to us (which seems to trigger the problem), and was thinking it was possibly because the read / write on one thread happened before we were finished the onWritePossible call at the container level.
"From experience, timing issues tend to be triggered by non-container threads.": Yes, but the fact that doWrite() was moved out of the sync block to fix a deadlock situation indicates otherwise. Ping on every call fixes the problem for us, and we will be working to move to 8.5 soon anyway. Thanks, -Rob On Thu, Mar 30, 2017 at 1:09 PM, Mark Thomas <ma...@apache.org> wrote: > On 29/03/17 04:04, Robert Lewis wrote: > > Thanks Mark. I will take a look at the test you linked in (seems like > Clint > > already is). > > > > I have a question regarding your previous note "The short version is that > > it is possible that there are two threads". On 8.0.38, doWrite() sets > it's > > scoped handler and buffers to the class level instance, then proceeds to > > call onWritePossible(true). onWritePossible creates a new local variable > > that seems like it copies the class level buffers state (not sure if this > > is a deep clone or not, i'll have to test this), but it does not > replicate > > the handler reference. If onWritePossible() is busy working writing the > > buffers to the socket and another thread calls doWrite(), that class > level > > state will be swapped before onWritePossible is finished, resulting in a > > possible race condition for a swapped out buffer or handler. > > > > Not claiming to know the code as well as those who maintain it, but it > > would be nice to know if someone thinks this could happen resulting in > > unexpected behavior (eg. we are writing to a socket and the the read > thread > > responds with a write on another thread). > > There are checks much higher up the stack that should throw an > IllegalStateException (from memory) if you try and write two messages at > once. > > From experience, timing issues tend to be triggered by non-container > threads. A typical pattern is that the first thread has some unexpected > side-effects that impact on the second thread - often because of > unexpected execution order. > > > Curious why onWritePossible(true) is not called with the doWrite() state? > > (eg. onWritePossible(true, buffers, handler))? > > The callers don't all have that information. > > > This is just an observation at this point so looking for an opinion on > > whether or not something described above could happen. If what was stated > > (that a write and a read / response write) can operate on two threads, I > > don't see why not. > > > > Continuing to run off and play with the tests you have linked. > > Again from experience unless you have some clues to point you in the > right direction, finding these issues via code inspection is hard. > Equally, a reproducible test case is rare. What tends to happen is a > more iterative approach that starts with an occasionally failing test > case and a combination of debugging, logging and extra debug statements > is used to narrow down the location of the problem. > > Mark > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
