GitHub user revans2 opened a pull request:
https://github.com/apache/storm/pull/750
STORM-350: Upgrade to newer version of disruptor
This upgrades to version 3.3.2 of the Disruptor Queue. There have been
questions about stability in the past, and also out of order delivery.
I really wanted to be sure that everything would be about the same. I ran
the DisruptorQueue related unit tests over the weekend and got no failures at
all, with well over 10,000 runs.
I did some performance tests too using the FastWordCountTopology I added as
a part of this. I ran 5 times with the original 0.11.0-SNAPSHOT this is based
off of (e85921035fa9bb59d25f0347dc6d26002aac9fab) and with this branch. By
setting the topology.max.spout.pending to 200 I got essentially identical
results. The new Queue was slightly faster but it was small enough it could
just be noise.
Similarly when I did not set topology.max.spout.pending and relied on the
automatic throttling I got very similar numbers, although the variance between
the runs was much higher so having a real comparison there is much more
difficult.
@HeartSaVioR in the past you did some testing to see if out of order
delivery was happening, I would love it if you could take a look at this patch
and test it similarly.
Anecdotally I have seen this version behave better than the current one we
are using. I have seen no NPEs from tuples disappearing and I have seen that
show up in some of my stress testing using the old queue. Again I don't know
how often this happens so I cannot guarantee that it was a disruptor bug.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/revans2/incubator-storm disruptor-upgrade
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/750.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #750
----
commit 46df80cfc63f01c3a5587a879ce0d721fad3f07f
Author: Robert (Bobby) Evans <[email protected]>
Date: 2015-09-18T21:33:39Z
STORM-350: Upgrade to newer version of disruptor
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
