GitHub user roshannaik opened a pull request:
https://github.com/apache/storm/pull/2502
new PR for STORM-2306
Since the [old PR page](https://github.com/apache/storm/pull/2241) had
become unusable (due to lots of old comments). Creating a new PR for the same.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/roshannaik/storm STORM-2306-2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/2502.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2502
----
commit 25cce136d256ea07183d6503246abe6c211ac522
Author: Roshan Naik <roshan@...>
Date: 2017-07-25T03:01:00Z
Messaging susbsytem redesign. Rebased to latest master. Validated
compilation and few simple topo runs. C->N: 8 mil/sec. 2.8 mil/sec with 2
workers. 1 mil/sec with ack (30-800 micosec). C->ID->N: 6.2mill/sec.
commit 4153a2f6fb5036a216b285ca00c3cce8656f9996
Author: Roshan Naik <roshan@...>
Date: 2017-07-26T03:34:48Z
addressing satish review comments
commit 2f070c30d09469a08edcb752ebdece61f4abddab
Author: Roshan Naik <roshan@...>
Date: 2017-08-01T07:39:10Z
addressing review comments
commit ec4430616428d79c852e5a6a1fe0c64c61c4d021
Author: Roshan Naik <roshan@...>
Date: 2017-08-02T06:41:35Z
Added Bolt Sleep strategy
commit 4a76691e744e16de9a58a44e359330856373da79
Author: Roshan Naik <roshan@...>
Date: 2017-08-03T19:38:42Z
Added BackPressure Sleep Strategy. Shutdown bug fix
commit b052564ac7b66537815b7a0137a558f2927ddf98
Author: Roshan Naik <roshan@...>
Date: 2017-08-04T03:56:12Z
Changing defaults for tighter latency and favor lower/medium throughput
topologies
commit 8eea6aa0457c280ad582a3112b6828b6c9e60a00
Author: Roshan Naik <roshan@...>
Date: 2017-08-08T03:29:31Z
Some minor simplifications. overriding settings within storm-perf topos as
needed
commit 300a4f84eed239c08ea1f2ac0c4945353f0555e6
Author: Roshan Naik <roshan@...>
Date: 2017-08-08T03:30:17Z
resolving runtime exception issues due to conflict with STORM-2672
commit 23b5f0979f44f157c0634296bd311ad6b290e276
Author: Robert Evans <evans@...>
Date: 2017-08-08T15:44:14Z
Merge branch 'STORM-2306m' of https://github.com/roshannaik/storm into
STORM-2306
This is a test, this is only a test
commit 1c628d1ba1eeea81e21b5d59a5cb5eaabea6c03b
Author: Robert Evans <evans@...>
Date: 2017-08-08T20:44:51Z
FIX some of the tests and metrics for the system bolt
commit b75192e1ae0311747bf00b5a6d320fc0e662183f
Author: roshannaik <roshan@...>
Date: 2017-08-11T20:07:03Z
Merge pull request #3 from revans2/STORM-2306
FIX some of the tests and metrics for the system bolt
commit 2506c6eaa9f04f3c57e6875d6078135ac99b8163
Author: Roshan Naik <roshan@...>
Date: 2017-08-12T00:37:58Z
System Bolt doesnt get much incoming traffic, so can use its own
pre-defined sleep strategy (consider making this configurable if useful)
commit 47579ea936cde21f87973ad4b73226737bfe806e
Author: Roshan Naik <roshan@...>
Date: 2017-08-15T09:01:09Z
Fixing process_latency bug reported by Bobby
commit 4704245e44b0868cfa5237f97a1fca0686df11a9
Author: Roshan Naik <roshan@...>
Date: 2017-08-18T03:33:55Z
Fixing the bug of wrong conf used for flushing as reported by Bobby. Adding
Perf tuning document
commit 5cb42cbb247052eac1c7858192db09a583921258
Author: Roshan Naik <roshan@...>
Date: 2017-08-19T00:41:20Z
Bug fix in BackPressure mode WaitStrategy to use the right settings.
Updates to Performance.md
commit 942fbe6fd60fffe9d1cdc08e960d4b7752fc69ad
Author: Roshan Naik <roshan@...>
Date: 2017-08-21T21:35:29Z
Aruns fix for OOM issue in Netty code (workerTransfer)
commit 5c223be2f774de9a9f4dded8450ef181f897a67a
Author: Roshan Naik <roshan@...>
Date: 2017-08-22T03:24:56Z
Flush Tuples need to be put into the Worker Transfer Queue as well
commit b1a99a038319dfe15d4ac4dfde819f0008ce5291
Author: Roshan Naik <roshan@...>
Date: 2017-08-28T23:11:40Z
Allow ConstSpout to have a configurable sleep for throttling. Improve
defaults for sleep strategy, queue size & max.spout.pending.
commit f9beb5358c91a799bec49f071a8f68153042a48a
Author: Roshan Naik <roshan@...>
Date: 2017-08-29T05:29:31Z
Revert changes in ShuffleGrouping until there is consensus on thread safety
in groupers.
commit 6cd1424956a0d95f36a6b10158107dfd4ac02a1b
Author: Roshan Naik <roshan@...>
Date: 2017-08-31T10:54:24Z
Introduce overflow to avoid deadlocked cycles involving ACKer. Added more
non-blocking methods to JCQueue. Reverting max.spout.pending to null. Minor
updates to ConstSpoutNullBolt Topo. Noticed a bug (not fixed) that can lead to
OOM in multiworker mode
commit 6bfda34cfdc945cd0cb58c7e13369d577129e6b2
Author: roshannaik <roshan@...>
Date: 2017-08-31T11:03:59Z
Merge branch 'master' into STORM-2306m
commit ece4ce2d8c829da53cfb48448d4f988110e237aa
Author: Roshan Naik <roshan@...>
Date: 2017-09-01T21:10:34Z
bug fix in SpoutOutputCollectorImpl.emit()
commit ba0b9e2a2a5d5a99c50f2506a7f76a261fa19fc0
Author: Roshan Naik <roshan@...>
Date: 2017-09-14T07:47:00Z
Moving worker xsfer code out of WorkerState into WorkerTransfer.java -
interworker=3mill/sec. withAck=700k/s,1.3ms
commit c275bfc11c12ff743ccf26c5bbc8db08e7c34030
Author: Roshan Naik <roshan@...>
Date: 2017-11-14T23:27:31Z
- Squishing recent commits related to Interworker backpressure communication
- Fixed some issues found during multi-worker testing
- Fixes to perf topos to pick up cmd line args correctly
- Removing support for single producer mode in JCQueue
- Almost all msg transfers are non-blocking now (other than metrics and
credentials-change notifications). Allows spouts and bolts to processes metrics
ticks even if output path is choked due to backpressure
- Some temporary debug logs that need to be removed after testing
- Needs more multiworker mode testing
commit a6f84c6784eb1a66b968b6b01262e76a136f3d89
Author: Roshan Naik <roshan@...>
Date: 2017-11-30T10:19:33Z
- Fixing bug in tryTransferLocal() that caused Trident issues.
- Adding workerID to BpStatus for better debuggability
- Logging the length of an idle stretch for BPÂ & max.spout.pending wait
situations
- Changes to defaults:Â topology.executor.receive.buffer.size=32k
(rounding up to power of 2),  topology.flush.tuple.freq.millis=1 (same as
master)
- minor fixes and improvements
commit d0aa1ad27b9375a2c393c760427e24f3158441b8
Author: Roshan Naik <roshan@...>
Date: 2017-11-30T10:21:52Z
Change logLevels. Fix unit test
commit b6fc2b8aaa92c41f786b53039989b373cbf6cd1d
Author: Roshan Naik <roshan@...>
Date: 2017-12-03T09:08:35Z
Addressing minor TODOs
commit 39b395d4f5b3c5fa50a1ebdb08333264d40a9308
Author: Roshan Naik <roshan@...>
Date: 2017-12-05T01:12:21Z
- Renaming config settings for easier understanding ( Progressive Wait
Strategy settings + topology.flush.tuple.freq.millis ->
topology.batch.flush.interval.millis )
commit 1f35bc9c16795177704ffd8eed4500aa735c24e3
Author: Roshan Naik <roshan@...>
Date: 2017-12-05T01:13:34Z
- Added message Drop Metrics
- Added BackPressure unit tests
commit b179a1937ec46ac0651c88b15b9e35dbfd55aef6
Author: Roshan Naik <roshan@...>
Date: 2017-12-05T05:05:29Z
Client.send throws IOException is connection closes. Allows it to recover
from remote worker dying
----
---
