GitHub user Parth-Brahmbhatt opened a pull request: https://github.com/apache/storm/pull/354
Storm 166: Nimbus HA design doc and implementation. I have deleted the bit torrent implementation from this pull request as the only available bit torrent library does not support tracker less torrents. In absence of tracker less torrents a single tracker becomes a single point of failure and a multi tracker implementation requires that if a tracker host fails the replacement host has same dns/network configuration. Some manual tests I executed: * start 3 nimbuses, test simple word count topology works. try storm list/activate/deactivate/rebalance/kill from ui and CLI. * set the replication factor to 2 run the first test again. * bring up a new nimbus, ensure it catches up and competes for leader lock. * with 3 nimbuses and 2 topologies, delete one topology code from each non leader nimbus. After killing master nimbus, ensure one of them eventually becomes leader. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Parth-Brahmbhatt/incubator-storm STORM-166 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/354.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #354 ---- commit 2b8dade2fe3fd000008e2ef217b04fba7ab327ef Author: Midpoint Applications <dl-kestrelmidpointapplicati...@exchange.boeing.com> Date: 2014-07-31T18:58:34Z upgraded to logback 1.0.13 commit 4e1f474e5b33fca708c4f8a9c3124cbf1f55611e Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: 2014-09-12T21:24:16Z Nimbus-HA: initial commit with leader election code. commit 670ad01faf6818f5606b98241811db878ac7ad53 Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: 2014-09-12T21:25:54Z removing duplicate dependency declaration. commit 9280afb51d6a59088520c0b4b0d46d1cb5824d13 Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: 2014-09-14T19:01:24Z Adding leader election nimbus test case. commit d0aa8ffae10c5b7795e7db22d22d21f61726ccdc Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: 2014-09-22T17:56:24Z Adding the code distribution interface. commit 4cb2eeee4cfec72a638ba149e5aa5cda725d5c9d Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: 2014-09-23T15:30:06Z Moving the bitTorrent code to ICodeDistributor interface. commit 4d0c6500b59e4b7cb74cc37b5849ae041092a59b Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: 2014-09-24T20:12:38Z adding back the deleted superviosr test. commit 50c4c346afa1eb04d0ebeb57aec40b47ad663dce Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: 2014-10-06T17:59:21Z Merge remote-tracking branch 'upstream/master' into STORM-166 commit f5ac420ba240be888e7c65d4fb5955923b0a2a38 Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: 2014-10-07T22:09:13Z Adding HDFSCodeDistributor. commit b074843ef4dc8e8f987071c42a12ad0c6c7b120d Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: 2014-10-07T22:16:45Z Merge remote-tracking branch 'upstream/master' into STORM-166 Conflicts: storm-core/test/clj/backtype/storm/supervisor_test.clj commit 0e24a4363c57a6aa4e994fc3b42fe347b3e58b9e Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: 2014-10-08T18:05:06Z Fixing the supervisor test failures. commit 050e8fd72c7aef8017a2c2d1ff9b95d5095544a1 Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: 2014-10-08T22:50:14Z BugFix: bittorrent code was downloaded in wrong folder. Fixed in hecky way. All tests pass now and was able to run wordcount and excalmation topologies. commit 932cecfc8b721d63e222e920aba377975c8ae89b Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: 2014-10-09T16:22:43Z Making the code distributor injectable via strom configs. Changes to HDFSCodeDistributor. Working version of HDFSCodeDistributor. commit 16a3ce4a071f0a40ccb34de48c3e21830cdfacad Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: 2014-10-22T00:56:20Z Adding the auto discovery of nimbuses for all clients. Modified UI to display list of nimbus hosts and current leader. commit fbb326be7d6a105a7ce40ca06cd3937f9da9d138 Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: 2014-10-22T00:57:01Z Merge remote-tracking branch 'upstream/master' into STORM-166 commit f0701b333716cc6368daeee9f9c5a11383257293 Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: 2014-10-22T02:03:09Z Removing NIMBUS_HOST config values, updated unit tests so they can succeed in absence of the NIMBUS-CONF value. commit 046262abd2d796a2056f7ce9c49294beacd0849c Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: 2014-10-22T20:07:25Z Add a meaningful message when no nimbus participant is found. commit 11aadf5201800685ee48147af7b78eec9181df1f Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: 2014-12-01T20:03:18Z Removing unused import. commit 728b35a966df9be45ba60213addce431fbbfe840 Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: 2014-12-01T21:29:39Z Cleaning up torrent download directory. commit 771f6a9b861afa605e9386fbac9ae0f598b77039 Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: 2014-12-01T23:22:53Z Implemented leader latch listener that relinquishes lock if it does not have all the active topology code locally, keeps the lock if it can verify all active topology code exists locally. commit 609f569b1a779ef0db7be5e957c48107bede1560 Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: 2014-12-02T17:25:06Z Removing todos from HDFDCodeDistributor. commit 50a33ac19ebba3b22c5c07d054c00080781004a4 Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: 2014-12-02T22:34:11Z Fixing type in log statement. commit 522696cbfa0b5543fb8812e0ecb0acf7bfffef93 Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: 2014-12-02T22:35:35Z Adding wait loop in master nimbus to allow for desired code replication to be achieved before the topology is activated. commit e037d3b665750b7e23ffdf13753eb40dedf74ed8 Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: 2014-12-02T22:35:59Z Removing unwanted TODO. commit e5c14e2a8a27a5342e852201507e79258a3201dd Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: 2014-12-02T23:26:19Z adding logging statements to wait for replication method. commit c684939ee75cfd6b7dd6e9b1cfa19c5925f7d468 Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: 2014-12-03T19:57:00Z Adding cluster state for code-distributor. commit 32d78388fba9a20b2ba68a3751180c698bac7f65 Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: 2014-12-05T22:03:09Z Making zkLeaderElector addToLeaderQueue/removeFromLeaderQueue idempotent commit dda2ee1436c7a444c3f50d00dfabc7214342bd70 Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: 2014-12-05T22:06:07Z Addig nimbus sync-code background thread. commit 4e35c1a0820009c005a3b7c30122d04a9197934c Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: 2014-12-08T17:50:41Z making downloads atomic. commit d1982427b4cab805509f993596171346d6759fe5 Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: 2014-12-08T17:52:11Z Binding the bittorrent to all network interfaces. ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---