Hi Eric, I can reproduce your problem although I have no idea yet what is going wrong. Do you have any updates ?
Regards, Manu Zhang On Mon, Aug 8, 2016 at 11:03 AM Eric Newton <eric.new...@gmail.com> wrote: > I wrote a little unit test and it works as I expect on linux (ubuntu, > redhat, openjdk 1.8). > > It fails regularly on a colleagues' OSX build environment with Oracle > jdk1.8.0_45. > > The intent of the test is to write out a file of random lines. Each line is > sent into a kafka as a message. A single consumer reads these messages and > writes them to a file. If the consumer waits longer than a second receiving > data, it assumes that all the messages have been delivered, and closes the > output file; the test compares the input and output files. > > On OSX, the output file is always short. > > 1) am I making unreasonable assumptions in the test? I know that its legal > for the consuming thread to fall asleep for a second, or for kafka to wait > and not deliver the message for a second, but is it reasonable? There's > nothing else going on, no swapping, plenty of free disk and memory. > > 2) are there known differences in OS/JVM behaviors that might affect the > test? > > 3) we have experimented with longer waits for all the messages, but have > not found anything that works 100% of the time. I am concerned that my > basic understanding of how kafka is supposed to work is fundamentally > flawed. > > For example, does kafka not provide all available messages unless a buffer > size is met? Did I miss some flush or commit call that is required to see > all messages? Is there a setting to reduce the wait time for the last > messages? > > I think apache strips out attachments, so I've made the code available here > <https://drive.google.com/open?id=0ByzSB9b42jf0aDZYRnNJdWx1OHc> as a > simple > stand-alone maven package (.tar.gz). Unpack, review, and run with `mvn > verify`. > > Thanks! > > -Eric >
