I actually didn't realize that HDDs had that performance limitation, so thanks for that info, and it now makes sense why in certain circumstances (i.e. where messages are more closely spaced than 10ms/msg) there could be a throughput improvement by batching. Keep in mind though that if you do 10ms batches, you're going to add an average of 5ms latency to your messages compared to the theoretical throughput of a SSD that didn't have that limitation, even if your current message rates are slow enough that the SSD can keep up without batching.
So if batching is implemented, I'd definitely want it to be a configurable option rather than the one and only way it works. The easy option would simply be a configuration option to turn batching on or off (and always pay the 5ms latency cost); the better option would be an adaptive strategy that (if enabled) recognizes when you throughput crosses the worth-it threshold and begins batching, and then recognizes when you drop back below it and goes back to the sync-per-message approach. Also, keep in mind that this strategy is only going to make a performance improvement if it's possible to process messages in parallel; if the code is written so you can't move to the next message in a destination till the previous one is sync'ed, then you'll get an improvement by eliminating contention between destinations but you'll go the same speed (slower, actually, due to the 5ms average latency of the batching) within a single destination. So as usual, YMMV, which is another reason why this should be an option rather than the only option. Tim On Sat, Mar 14, 2015 at 12:35 PM, Kevin Burton <bur...@spinn3r.com> wrote: > > So if you batch disk writes, you either batch acks (which has a negative > > performance impact; I'm not sure if the loss will be greater or less than > > the gain from batching writes, but at a minimum it'll decrease the > expected > > improvement from batching writes) > > > > Yes. This is the point I’m trying to make. > > If you’re on HDD without any form of write caching, the MAX you can commit > is around 100 sync’s per second. > > So the sync throughput for ActiveMQ is lower than this due to ActiveMQ > needing to do additional work (memory copying, etc). But in theory it’s > probably 95% of this number. > > However, if you batch commit, you have 10ms between each sync where you can > accumulate messages. ALL the messages you accumulate in that window can > then be sync’d in the next round. > > (I’m sure you guys know this stuff, just codifying it) > > The next time around you can write ALL the message at once, and with > LevelDB the performance should be near sequential disk IO (in theory, > since it’s writing to the log) which is pretty darn fast. > > So in practice you DO have to batch the ACKs back but that’s more than made > up for with the performance bump. > > Disk is just so amazingly slow. > > I’d be willing to do a proof of concept branch with benchmarks if someone > can point me to the right spot in the code. Should be a small chunk of > code. > > or you optimistically ack before the bits > > hit the disk (which violates the JMS contract because it allows message > > loss if the server crashes before the batch is written). > > > > > > -- > > Founder/CEO Spinn3r.com > Location: *San Francisco, CA* > blog: http://burtonator.wordpress.com > … or check out my Google+ profile > <https://plus.google.com/102718274791889610666/posts> > <http://spinn3r.com> >
