GitHub user Parth-Brahmbhatt opened a pull request:
https://github.com/apache/storm/pull/354
Storm 166: Nimbus HA design doc and implementation.
I have deleted the bit torrent implementation from this pull request as the
only available bit torrent library does not support tracker less torrents. In
absence of tracker less torrents a single tracker becomes a single point of
failure and a multi tracker implementation requires that if a tracker host
fails the replacement host has same dns/network configuration.
Some manual tests I executed:
* start 3 nimbuses, test simple word count topology works. try storm
list/activate/deactivate/rebalance/kill from ui and CLI.
* set the replication factor to 2 run the first test again.
* bring up a new nimbus, ensure it catches up and competes for leader lock.
* with 3 nimbuses and 2 topologies, delete one topology code from each non
leader nimbus. After killing master nimbus, ensure one of them eventually
becomes leader.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Parth-Brahmbhatt/incubator-storm STORM-166
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/354.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #354
----
commit 2b8dade2fe3fd000008e2ef217b04fba7ab327ef
Author: Midpoint Applications
<[email protected]>
Date: 2014-07-31T18:58:34Z
upgraded to logback 1.0.13
commit 4e1f474e5b33fca708c4f8a9c3124cbf1f55611e
Author: Parth Brahmbhatt <[email protected]>
Date: 2014-09-12T21:24:16Z
Nimbus-HA: initial commit with leader election code.
commit 670ad01faf6818f5606b98241811db878ac7ad53
Author: Parth Brahmbhatt <[email protected]>
Date: 2014-09-12T21:25:54Z
removing duplicate dependency declaration.
commit 9280afb51d6a59088520c0b4b0d46d1cb5824d13
Author: Parth Brahmbhatt <[email protected]>
Date: 2014-09-14T19:01:24Z
Adding leader election nimbus test case.
commit d0aa8ffae10c5b7795e7db22d22d21f61726ccdc
Author: Parth Brahmbhatt <[email protected]>
Date: 2014-09-22T17:56:24Z
Adding the code distribution interface.
commit 4cb2eeee4cfec72a638ba149e5aa5cda725d5c9d
Author: Parth Brahmbhatt <[email protected]>
Date: 2014-09-23T15:30:06Z
Moving the bitTorrent code to ICodeDistributor interface.
commit 4d0c6500b59e4b7cb74cc37b5849ae041092a59b
Author: Parth Brahmbhatt <[email protected]>
Date: 2014-09-24T20:12:38Z
adding back the deleted superviosr test.
commit 50c4c346afa1eb04d0ebeb57aec40b47ad663dce
Author: Parth Brahmbhatt <[email protected]>
Date: 2014-10-06T17:59:21Z
Merge remote-tracking branch 'upstream/master' into STORM-166
commit f5ac420ba240be888e7c65d4fb5955923b0a2a38
Author: Parth Brahmbhatt <[email protected]>
Date: 2014-10-07T22:09:13Z
Adding HDFSCodeDistributor.
commit b074843ef4dc8e8f987071c42a12ad0c6c7b120d
Author: Parth Brahmbhatt <[email protected]>
Date: 2014-10-07T22:16:45Z
Merge remote-tracking branch 'upstream/master' into STORM-166
Conflicts:
storm-core/test/clj/backtype/storm/supervisor_test.clj
commit 0e24a4363c57a6aa4e994fc3b42fe347b3e58b9e
Author: Parth Brahmbhatt <[email protected]>
Date: 2014-10-08T18:05:06Z
Fixing the supervisor test failures.
commit 050e8fd72c7aef8017a2c2d1ff9b95d5095544a1
Author: Parth Brahmbhatt <[email protected]>
Date: 2014-10-08T22:50:14Z
BugFix: bittorrent code was downloaded in wrong folder. Fixed in hecky way.
All tests pass now and was able to run wordcount and excalmation topologies.
commit 932cecfc8b721d63e222e920aba377975c8ae89b
Author: Parth Brahmbhatt <[email protected]>
Date: 2014-10-09T16:22:43Z
Making the code distributor injectable via strom configs. Changes to
HDFSCodeDistributor. Working version of HDFSCodeDistributor.
commit 16a3ce4a071f0a40ccb34de48c3e21830cdfacad
Author: Parth Brahmbhatt <[email protected]>
Date: 2014-10-22T00:56:20Z
Adding the auto discovery of nimbuses for all clients. Modified UI to
display list of nimbus hosts and current leader.
commit fbb326be7d6a105a7ce40ca06cd3937f9da9d138
Author: Parth Brahmbhatt <[email protected]>
Date: 2014-10-22T00:57:01Z
Merge remote-tracking branch 'upstream/master' into STORM-166
commit f0701b333716cc6368daeee9f9c5a11383257293
Author: Parth Brahmbhatt <[email protected]>
Date: 2014-10-22T02:03:09Z
Removing NIMBUS_HOST config values, updated unit tests so they can succeed
in absence of the NIMBUS-CONF value.
commit 046262abd2d796a2056f7ce9c49294beacd0849c
Author: Parth Brahmbhatt <[email protected]>
Date: 2014-10-22T20:07:25Z
Add a meaningful message when no nimbus participant is found.
commit 11aadf5201800685ee48147af7b78eec9181df1f
Author: Parth Brahmbhatt <[email protected]>
Date: 2014-12-01T20:03:18Z
Removing unused import.
commit 728b35a966df9be45ba60213addce431fbbfe840
Author: Parth Brahmbhatt <[email protected]>
Date: 2014-12-01T21:29:39Z
Cleaning up torrent download directory.
commit 771f6a9b861afa605e9386fbac9ae0f598b77039
Author: Parth Brahmbhatt <[email protected]>
Date: 2014-12-01T23:22:53Z
Implemented leader latch listener that relinquishes lock if it does not
have all the active topology code locally, keeps the lock if it can verify all
active topology code exists locally.
commit 609f569b1a779ef0db7be5e957c48107bede1560
Author: Parth Brahmbhatt <[email protected]>
Date: 2014-12-02T17:25:06Z
Removing todos from HDFDCodeDistributor.
commit 50a33ac19ebba3b22c5c07d054c00080781004a4
Author: Parth Brahmbhatt <[email protected]>
Date: 2014-12-02T22:34:11Z
Fixing type in log statement.
commit 522696cbfa0b5543fb8812e0ecb0acf7bfffef93
Author: Parth Brahmbhatt <[email protected]>
Date: 2014-12-02T22:35:35Z
Adding wait loop in master nimbus to allow for desired code replication to
be achieved before the topology is activated.
commit e037d3b665750b7e23ffdf13753eb40dedf74ed8
Author: Parth Brahmbhatt <[email protected]>
Date: 2014-12-02T22:35:59Z
Removing unwanted TODO.
commit e5c14e2a8a27a5342e852201507e79258a3201dd
Author: Parth Brahmbhatt <[email protected]>
Date: 2014-12-02T23:26:19Z
adding logging statements to wait for replication method.
commit c684939ee75cfd6b7dd6e9b1cfa19c5925f7d468
Author: Parth Brahmbhatt <[email protected]>
Date: 2014-12-03T19:57:00Z
Adding cluster state for code-distributor.
commit 32d78388fba9a20b2ba68a3751180c698bac7f65
Author: Parth Brahmbhatt <[email protected]>
Date: 2014-12-05T22:03:09Z
Making zkLeaderElector addToLeaderQueue/removeFromLeaderQueue idempotent
commit dda2ee1436c7a444c3f50d00dfabc7214342bd70
Author: Parth Brahmbhatt <[email protected]>
Date: 2014-12-05T22:06:07Z
Addig nimbus sync-code background thread.
commit 4e35c1a0820009c005a3b7c30122d04a9197934c
Author: Parth Brahmbhatt <[email protected]>
Date: 2014-12-08T17:50:41Z
making downloads atomic.
commit d1982427b4cab805509f993596171346d6759fe5
Author: Parth Brahmbhatt <[email protected]>
Date: 2014-12-08T17:52:11Z
Binding the bittorrent to all network interfaces.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
