[ https://issues.apache.org/jira/browse/STORM-876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15007401#comment-15007401 ]
ASF GitHub Bot commented on STORM-876: -------------------------------------- GitHub user redsanket reopened a pull request: https://github.com/apache/storm/pull/845 [STORM-876] Blobstore API Please take a look at the blobstore API. The design docs have been uploaded earlier under the same JIRA. Currently, the blobstore API also works with Nimbus-HA. Apart from the unit tests written for blobstore and nimbus leader election, I have performed few manual tests. Having the PR early looking for suggestions and any missed failover scenarios while the work continues on other additional integration tests and better interface. The manual tests that have been performed are 1. CRUD operations for the blobstore API. Tested for the Local and Hdfs blobstore operations to work with Nimbus-HA and whether the other nimbodes mirror the operations performed on master nimbus. 2. Brought up 3 nimbodes and launched a word count topology. Killed the leader daemon and checked whether other non-leader takes up the leadership and continues to run. 3. Tested whether topology waits for replication to take place before it is launched. For hdfs blobstore the blobstore.replication should be set greater than or equal to topology.min.replication to work effectively. 4. The blobstore with nimbus-ha does not guarantee updates to sync effectively in the case of failover. Follow up code to be written. 1. Integration test for blobstore with nimbus-ha 2. Move the sync thread in nimbus to blobstore. Waiting on pacemaker push back as it uses some of its code and write the callback code for blobstore You can merge this pull request into a Git repository by running: $ git pull https://github.com/redsanket/storm blobstore Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/845.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #845 ---- commit 6e1ee30ed6e142033a8b3f264918806630da1bf1 Author: Sanket <schintap@untilservice-lm> Date: 2015-11-02T19:59:13Z Squashed commit of the following: commit 3403a6da661f5ff002cf5e4571853f9b95d8e3c8 Author: Sanket <schintap@untilservice-lm> Date: Mon Nov 2 13:38:14 2015 -0600 resolved unit test errors caused due to merge commits commit 898c1d64b5dcbcca2d1efc9a78e731316c1292c0 Merge: 9909a50 8ba776b Author: Sanket <schintap@untilservice-lm> Date: Mon Nov 2 11:41:13 2015 -0600 resolved merge conflicts commit 9909a50a8b1037e16b21f45ee88c8237f33bd7b8 Author: Sanket <schintap@untilservice-lm> Date: Mon Nov 2 10:57:29 2015 -0600 cleaned-up debug statements commit 3f2e1796a74799702113303e896d353ce4218c29 Author: Sanket <schintap@untilservice-lm> Date: Fri Oct 30 23:25:21 2015 -0500 zookeeper state code and code clean up commit ac4db652aaad8f3ec71aca50eb31579bf90bd887 Author: Sanket <schintap@untilservice-lm> Date: Thu Oct 29 11:26:17 2015 -0500 made changes to the sync blob code to fix race condition commit c96ffcaac5d247da1c77a1e1699b2d7b7f76ebf5 Author: Sanket <schintap@untilservice-lm> Date: Tue Oct 27 13:38:17 2015 -0500 resolved unit test errors commit f8f7ebab8cb636a296d84b7e719c368e988063c8 Author: Sanket <schintap@untilservice-lm> Date: Mon Oct 26 22:51:11 2015 -0500 added callback function commit c5f02cd54ef50039a6f4dde8f073e839b9ef7b3a Author: Sanket <schintap@untilservice-lm> Date: Fri Oct 23 16:20:41 2015 -0500 merge commit, modified unit tests to suit blob-store commit fdc77d54054470fdaa419dbedaa0d69fdf1f63ce Merge: eb0ea1e f75fdde Author: Sanket <schintap@untilservice-lm> Date: Wed Oct 21 13:28:46 2015 -0500 modified blobstore to accomodate nimbus ha commit eb0ea1e167ebfee94e3a0911f05c2102cdeb77b8 Author: Sanket <schintap@untilservice-lm> Date: Wed Oct 21 13:08:22 2015 -0500 integrated blobstore code commit f25e7ba8139b0e2ed56ab532339d0b15b4563c6e Author: Sanket <schintap@untilservice-lm> Date: Thu Oct 15 19:06:52 2015 -0500 initial commit commit ba74039b93ee34067e2511e280a50e3958dc8741 Author: Sanket <schintap@untilservice-lm> Date: Thu Oct 8 15:04:42 2015 -0500 modified thrift files to include blobstore related classes commit 73859f70e21e4acde0b3b906b53786c7d2b7a5c5 Author: Sanket <schintap@untilservice-lm> Date: Thu Oct 8 15:01:46 2015 -0500 integrating blobstore and making sure it builds commit 2506665b02d8808efd20760f1a11e99239518496 Author: Sanket <schintap@untilservice-lm> Date: 2015-11-02T20:34:09Z removed log message and fixed a unit test commit e8093acbecdfc8893bd9ac571ee8e48f3d610e80 Author: Sanket <schintap@untilservice-lm> Date: 2015-11-03T03:32:33Z removed date change generated files commit 04458b8f8d3ecb022995281630c130da30dd7b28 Author: Sanket <schintap@untilservice-lm> Date: 2015-11-03T04:13:01Z indentation edit commit a26bdf8efd94d26ba65489491594a712381373c7 Author: Sanket <schintap@untilservice-lm> Date: 2015-11-03T04:16:06Z indentation edit commit e2fdb5e9f01cf0e58ed102839ff9a01e757cb290 Author: Sanket <schintap@untilservice-lm> Date: 2015-11-03T05:05:33Z removed a commented method commit 73a5d2f9fcab0063cabf39ff9e7ba4486e1e9070 Author: Sanket <schintap@untilservice-lm> Date: 2015-11-03T05:08:05Z removed an unnecessary commit commit 435913a8325cd6c6085beba90afebae97bc29e3e Author: Sanket <schintap@untilservice-lm> Date: 2015-11-03T15:19:30Z removed few more date change generated files commit 87c89d561074e478a30f2ba611b068e2a05cfb91 Author: Sanket <schintap@untilservice-lm> Date: 2015-11-03T18:29:58Z indentation edits and formatting issues commit 8825b570d2e63eefdb04b2b72a681c22adaefb20 Author: Sanket <schintap@untilservice-lm> Date: 2015-11-03T21:01:24Z moved the createInZookeeper to command line and got rid of few indentation errors and space errors commit 114a2c14bad8ab3695b8f18c72813d343e56c8e9 Author: Sanket <schintap@untilservice-lm> Date: 2015-11-04T02:07:05Z addressed few more issues on indentation, formatting and documentation commit a1165a08237d6e639f3403ca340555be762cf54a Author: Sanket <schintap@untilservice-lm> Date: 2015-11-04T02:12:22Z addressed issues related to code optimization and removed comments from storm thrift as the design docs explain them commit 45837ecf26ad99019c890757d68f1a9fa6bd624b Author: Sanket <schintap@untilservice-lm> Date: 2015-11-04T15:49:17Z removed date change generated thrift files commit f1e0fbbada106a2b3adc9e0c9a0fed4ca01aa0a7 Author: Sanket <schintap@untilservice-lm> Date: 2015-11-04T16:12:59Z fixed to resolve single class imports commit f89d272fa7646e4715f209c46b581311fc3150f4 Author: Sanket <schintap@untilservice-lm> Date: 2015-11-04T20:07:07Z Added config validation map for blobstore map commit 73f02571728c88c37df14a5cbc78c7f0dd4541d3 Author: Sanket <schintap@untilservice-lm> Date: 2015-11-04T23:16:47Z removed Blob Replication struct commit 27aad4137e58f232f7ef8e69f44de7c79952d50e Author: Sanket <schintap@untilservice-lm> Date: 2015-11-05T00:40:53Z resolved few more indentation, spacing errors as per review commit 8691b7c0fbaa94ec5428844e008e08b434a9056c Author: Sanket <schintap@untilservice-lm> Date: 2015-11-05T16:30:09Z made some changes to the debug logs commit 650e386130a0506b909327dfc9a2ce3a00644901 Author: Sanket <schintap@untilservice-lm> Date: 2015-11-07T16:14:46Z added code to handle update scenarios by implementing a version generator commit 993c851e7ebbc4092f4b65d9b55ce1fe682af0c7 Author: Sanket <schintap@untilservice-lm> Date: 2015-11-08T05:09:41Z added few debug statements, refactoring and removed a bug found during updating blob commit 19d2b18ee99dfdd33741adae1e742167684d2f2a Author: Sanket <schintap@untilservice-lm> Date: 2015-11-08T21:55:03Z removed few debug comments, refactoring and handling few more failure scenarios commit 5d19faf59e76d5321ed0bb693189420e0debc635 Author: Sanket <schintap@untilservice-lm> Date: 2015-11-08T22:28:10Z merge commit commit 4a03bcd607de2fd894eb776227ba7c52ff57a898 Author: Sanket <schintap@untilservice-lm> Date: 2015-11-08T22:40:01Z added apache licenses and resolved a unchecked merge commit commit 115e25bbdea1e1ef671a42854f33a837953a24e1 Author: Sanket <schintap@untilservice-lm> Date: 2015-11-08T22:41:24Z resolved missed merge commit commit a49b47d132c8c9fe8f3e546d2a3b7a0730f2c732 Author: Sanket <schintap@untilservice-lm> Date: 2015-11-08T22:45:49Z resolved arity exception caused during merge commit 1e5e41f7b332d25c1fc793827549608de5f864bd Author: Sanket <schintap@untilservice-lm> Date: 2015-11-09T03:19:21Z resolved a unit test error caused by merge commit 32dff50a4d351477ec1854adc5c083fd48dc3f0f Author: Sanket <schintap@untilservice-lm> Date: 2015-11-09T15:45:41Z code clean up and exclude build directory from checking for license headers commit bae22c841f326d09125df6ab2492a1e6dbf0d12b Author: Sanket <schintap@untilservice-lm> Date: 2015-11-09T17:27:46Z removed unused import commit 32e5e791e8abc554e4413d38c4c229d797c58568 Author: Sanket <schintap@untilservice-lm> Date: 2015-11-09T18:55:46Z change to error log statements commit 8efd0ba9fb1307c38f3481f7ba2b0dd2aed90a17 Author: Sanket <schintap@untilservice-lm> Date: 2015-11-10T19:11:38Z merge commit ---- > Dist Cache: Basic Functionality > ------------------------------- > > Key: STORM-876 > URL: https://issues.apache.org/jira/browse/STORM-876 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core > Reporter: Robert Joseph Evans > Assignee: Robert Joseph Evans > Attachments: DISTCACHE.md, DistributedCacheDesignDocument.pdf > > > Basic functionality for the Dist Cache feature. > As part of this a new API should be added to support uploading and > downloading dist cache items. storm-core.ser, storm-conf.ser and storm.jar > should be written into the blob store instead of residing locally. We need a > default implementation of the blob store that does essentially what nimbus > currently does and does not need anything extra. But having an HDFS backend > too would be great for scalability and HA. > The supervisor should provide a way to download and manage these blobs and > provide a working directory for the worker process with symlinks to the > blobs. It should also allow the blobs to be updated and switch the symlink > atomically to point to the new blob once it is downloaded. > All of this is already done by code internal to Yahoo! we are in the process > of getting it ready to push back to open source shortly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)