[
https://issues.apache.org/jira/browse/STORM-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901295#comment-14901295
]
ASF GitHub Bot commented on STORM-350:
--------------------------------------
GitHub user revans2 opened a pull request:
https://github.com/apache/storm/pull/750
STORM-350: Upgrade to newer version of disruptor
This upgrades to version 3.3.2 of the Disruptor Queue. There have been
questions about stability in the past, and also out of order delivery.
I really wanted to be sure that everything would be about the same. I ran
the DisruptorQueue related unit tests over the weekend and got no failures at
all, with well over 10,000 runs.
I did some performance tests too using the FastWordCountTopology I added as
a part of this. I ran 5 times with the original 0.11.0-SNAPSHOT this is based
off of (e85921035fa9bb59d25f0347dc6d26002aac9fab) and with this branch. By
setting the topology.max.spout.pending to 200 I got essentially identical
results. The new Queue was slightly faster but it was small enough it could
just be noise.
Similarly when I did not set topology.max.spout.pending and relied on the
automatic throttling I got very similar numbers, although the variance between
the runs was much higher so having a real comparison there is much more
difficult.
@HeartSaVioR in the past you did some testing to see if out of order
delivery was happening, I would love it if you could take a look at this patch
and test it similarly.
Anecdotally I have seen this version behave better than the current one we
are using. I have seen no NPEs from tuples disappearing and I have seen that
show up in some of my stress testing using the old queue. Again I don't know
how often this happens so I cannot guarantee that it was a disruptor bug.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/revans2/incubator-storm disruptor-upgrade
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/750.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #750
----
commit 46df80cfc63f01c3a5587a879ce0d721fad3f07f
Author: Robert (Bobby) Evans <[email protected]>
Date: 2015-09-18T21:33:39Z
STORM-350: Upgrade to newer version of disruptor
----
> Update disruptor to latest version (3.2.1)
> ------------------------------------------
>
> Key: STORM-350
> URL: https://issues.apache.org/jira/browse/STORM-350
> Project: Apache Storm
> Issue Type: Improvement
> Reporter: Boris Aksenov
> Assignee: Boris Aksenov
> Priority: Minor
> Fix For: 0.10.0
>
> Attachments:
> 20141117-0.9.3-rc1-3-worker-separate-1-spout-and-2-bolts-failing-tuples.png,
> 20141117-0.9.3-rc1-one-worker-failing-tuples.png,
> 20141117-0.9.3-rc1-three-workers-1-spout-3-bolts-failing-tuples.png,
> 20141118-0.9.3-branch-3-worker-separate-1-spout-and-2-bolts-ok.png,
> 20141118-0.9.3-branch-one-worker-ok.png,
> 20141118-0.9.3-branch-three-workers-1-spout-3-bolts-ok.png, Storm UI1.pdf,
> Storm UI2.pdf, storm-0.9.3-rc1-failing-tuples.png,
> storm-0_9_2-incubating-failing-tuples.png,
> storm-0_9_2-incubating-no-failing-tuples.png,
> storm-failed-tuples-multi-node.png, storm-multi-node-without-350.png
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
