[
https://issues.apache.org/jira/browse/STORM-876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15007401#comment-15007401
]
ASF GitHub Bot commented on STORM-876:
--------------------------------------
GitHub user redsanket reopened a pull request:
https://github.com/apache/storm/pull/845
[STORM-876] Blobstore API
Please take a look at the blobstore API. The design docs have been uploaded
earlier under the same JIRA. Currently, the blobstore API also works with
Nimbus-HA. Apart from the unit tests written for blobstore and nimbus leader
election, I have performed few manual tests. Having the PR early looking for
suggestions and any missed failover scenarios while the work continues on other
additional integration tests and better interface. The manual tests that have
been performed are
1. CRUD operations for the blobstore API. Tested for the Local and Hdfs
blobstore operations to work with Nimbus-HA and whether the other nimbodes
mirror the operations performed on master nimbus.
2. Brought up 3 nimbodes and launched a word count topology. Killed the
leader daemon and checked whether other non-leader takes up the leadership and
continues to run.
3. Tested whether topology waits for replication to take place before it is
launched. For hdfs blobstore the blobstore.replication should be set greater
than or equal to topology.min.replication to work effectively.
4. The blobstore with nimbus-ha does not guarantee updates to sync
effectively in the case of failover.
Follow up code to be written.
1. Integration test for blobstore with nimbus-ha
2. Move the sync thread in nimbus to blobstore. Waiting on pacemaker push
back as it uses some of its code and write the callback code for blobstore
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/redsanket/storm blobstore
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/845.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #845
----
commit 6e1ee30ed6e142033a8b3f264918806630da1bf1
Author: Sanket <schintap@untilservice-lm>
Date: 2015-11-02T19:59:13Z
Squashed commit of the following:
commit 3403a6da661f5ff002cf5e4571853f9b95d8e3c8
Author: Sanket <schintap@untilservice-lm>
Date: Mon Nov 2 13:38:14 2015 -0600
resolved unit test errors caused due to merge commits
commit 898c1d64b5dcbcca2d1efc9a78e731316c1292c0
Merge: 9909a50 8ba776b
Author: Sanket <schintap@untilservice-lm>
Date: Mon Nov 2 11:41:13 2015 -0600
resolved merge conflicts
commit 9909a50a8b1037e16b21f45ee88c8237f33bd7b8
Author: Sanket <schintap@untilservice-lm>
Date: Mon Nov 2 10:57:29 2015 -0600
cleaned-up debug statements
commit 3f2e1796a74799702113303e896d353ce4218c29
Author: Sanket <schintap@untilservice-lm>
Date: Fri Oct 30 23:25:21 2015 -0500
zookeeper state code and code clean up
commit ac4db652aaad8f3ec71aca50eb31579bf90bd887
Author: Sanket <schintap@untilservice-lm>
Date: Thu Oct 29 11:26:17 2015 -0500
made changes to the sync blob code to fix race condition
commit c96ffcaac5d247da1c77a1e1699b2d7b7f76ebf5
Author: Sanket <schintap@untilservice-lm>
Date: Tue Oct 27 13:38:17 2015 -0500
resolved unit test errors
commit f8f7ebab8cb636a296d84b7e719c368e988063c8
Author: Sanket <schintap@untilservice-lm>
Date: Mon Oct 26 22:51:11 2015 -0500
added callback function
commit c5f02cd54ef50039a6f4dde8f073e839b9ef7b3a
Author: Sanket <schintap@untilservice-lm>
Date: Fri Oct 23 16:20:41 2015 -0500
merge commit, modified unit tests to suit blob-store
commit fdc77d54054470fdaa419dbedaa0d69fdf1f63ce
Merge: eb0ea1e f75fdde
Author: Sanket <schintap@untilservice-lm>
Date: Wed Oct 21 13:28:46 2015 -0500
modified blobstore to accomodate nimbus ha
commit eb0ea1e167ebfee94e3a0911f05c2102cdeb77b8
Author: Sanket <schintap@untilservice-lm>
Date: Wed Oct 21 13:08:22 2015 -0500
integrated blobstore code
commit f25e7ba8139b0e2ed56ab532339d0b15b4563c6e
Author: Sanket <schintap@untilservice-lm>
Date: Thu Oct 15 19:06:52 2015 -0500
initial commit
commit ba74039b93ee34067e2511e280a50e3958dc8741
Author: Sanket <schintap@untilservice-lm>
Date: Thu Oct 8 15:04:42 2015 -0500
modified thrift files to include blobstore related classes
commit 73859f70e21e4acde0b3b906b53786c7d2b7a5c5
Author: Sanket <schintap@untilservice-lm>
Date: Thu Oct 8 15:01:46 2015 -0500
integrating blobstore and making sure it builds
commit 2506665b02d8808efd20760f1a11e99239518496
Author: Sanket <schintap@untilservice-lm>
Date: 2015-11-02T20:34:09Z
removed log message and fixed a unit test
commit e8093acbecdfc8893bd9ac571ee8e48f3d610e80
Author: Sanket <schintap@untilservice-lm>
Date: 2015-11-03T03:32:33Z
removed date change generated files
commit 04458b8f8d3ecb022995281630c130da30dd7b28
Author: Sanket <schintap@untilservice-lm>
Date: 2015-11-03T04:13:01Z
indentation edit
commit a26bdf8efd94d26ba65489491594a712381373c7
Author: Sanket <schintap@untilservice-lm>
Date: 2015-11-03T04:16:06Z
indentation edit
commit e2fdb5e9f01cf0e58ed102839ff9a01e757cb290
Author: Sanket <schintap@untilservice-lm>
Date: 2015-11-03T05:05:33Z
removed a commented method
commit 73a5d2f9fcab0063cabf39ff9e7ba4486e1e9070
Author: Sanket <schintap@untilservice-lm>
Date: 2015-11-03T05:08:05Z
removed an unnecessary commit
commit 435913a8325cd6c6085beba90afebae97bc29e3e
Author: Sanket <schintap@untilservice-lm>
Date: 2015-11-03T15:19:30Z
removed few more date change generated files
commit 87c89d561074e478a30f2ba611b068e2a05cfb91
Author: Sanket <schintap@untilservice-lm>
Date: 2015-11-03T18:29:58Z
indentation edits and formatting issues
commit 8825b570d2e63eefdb04b2b72a681c22adaefb20
Author: Sanket <schintap@untilservice-lm>
Date: 2015-11-03T21:01:24Z
moved the createInZookeeper to command line and got rid of few indentation
errors and space errors
commit 114a2c14bad8ab3695b8f18c72813d343e56c8e9
Author: Sanket <schintap@untilservice-lm>
Date: 2015-11-04T02:07:05Z
addressed few more issues on indentation, formatting and documentation
commit a1165a08237d6e639f3403ca340555be762cf54a
Author: Sanket <schintap@untilservice-lm>
Date: 2015-11-04T02:12:22Z
addressed issues related to code optimization and removed comments from
storm thrift as the design docs explain them
commit 45837ecf26ad99019c890757d68f1a9fa6bd624b
Author: Sanket <schintap@untilservice-lm>
Date: 2015-11-04T15:49:17Z
removed date change generated thrift files
commit f1e0fbbada106a2b3adc9e0c9a0fed4ca01aa0a7
Author: Sanket <schintap@untilservice-lm>
Date: 2015-11-04T16:12:59Z
fixed to resolve single class imports
commit f89d272fa7646e4715f209c46b581311fc3150f4
Author: Sanket <schintap@untilservice-lm>
Date: 2015-11-04T20:07:07Z
Added config validation map for blobstore map
commit 73f02571728c88c37df14a5cbc78c7f0dd4541d3
Author: Sanket <schintap@untilservice-lm>
Date: 2015-11-04T23:16:47Z
removed Blob Replication struct
commit 27aad4137e58f232f7ef8e69f44de7c79952d50e
Author: Sanket <schintap@untilservice-lm>
Date: 2015-11-05T00:40:53Z
resolved few more indentation, spacing errors as per review
commit 8691b7c0fbaa94ec5428844e008e08b434a9056c
Author: Sanket <schintap@untilservice-lm>
Date: 2015-11-05T16:30:09Z
made some changes to the debug logs
commit 650e386130a0506b909327dfc9a2ce3a00644901
Author: Sanket <schintap@untilservice-lm>
Date: 2015-11-07T16:14:46Z
added code to handle update scenarios by implementing a version generator
commit 993c851e7ebbc4092f4b65d9b55ce1fe682af0c7
Author: Sanket <schintap@untilservice-lm>
Date: 2015-11-08T05:09:41Z
added few debug statements, refactoring and removed a bug found during
updating blob
commit 19d2b18ee99dfdd33741adae1e742167684d2f2a
Author: Sanket <schintap@untilservice-lm>
Date: 2015-11-08T21:55:03Z
removed few debug comments, refactoring and handling few more failure
scenarios
commit 5d19faf59e76d5321ed0bb693189420e0debc635
Author: Sanket <schintap@untilservice-lm>
Date: 2015-11-08T22:28:10Z
merge commit
commit 4a03bcd607de2fd894eb776227ba7c52ff57a898
Author: Sanket <schintap@untilservice-lm>
Date: 2015-11-08T22:40:01Z
added apache licenses and resolved a unchecked merge commit
commit 115e25bbdea1e1ef671a42854f33a837953a24e1
Author: Sanket <schintap@untilservice-lm>
Date: 2015-11-08T22:41:24Z
resolved missed merge commit
commit a49b47d132c8c9fe8f3e546d2a3b7a0730f2c732
Author: Sanket <schintap@untilservice-lm>
Date: 2015-11-08T22:45:49Z
resolved arity exception caused during merge
commit 1e5e41f7b332d25c1fc793827549608de5f864bd
Author: Sanket <schintap@untilservice-lm>
Date: 2015-11-09T03:19:21Z
resolved a unit test error caused by merge
commit 32dff50a4d351477ec1854adc5c083fd48dc3f0f
Author: Sanket <schintap@untilservice-lm>
Date: 2015-11-09T15:45:41Z
code clean up and exclude build directory from checking for license headers
commit bae22c841f326d09125df6ab2492a1e6dbf0d12b
Author: Sanket <schintap@untilservice-lm>
Date: 2015-11-09T17:27:46Z
removed unused import
commit 32e5e791e8abc554e4413d38c4c229d797c58568
Author: Sanket <schintap@untilservice-lm>
Date: 2015-11-09T18:55:46Z
change to error log statements
commit 8efd0ba9fb1307c38f3481f7ba2b0dd2aed90a17
Author: Sanket <schintap@untilservice-lm>
Date: 2015-11-10T19:11:38Z
merge commit
----
> Dist Cache: Basic Functionality
> -------------------------------
>
> Key: STORM-876
> URL: https://issues.apache.org/jira/browse/STORM-876
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-core
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: DISTCACHE.md, DistributedCacheDesignDocument.pdf
>
>
> Basic functionality for the Dist Cache feature.
> As part of this a new API should be added to support uploading and
> downloading dist cache items. storm-core.ser, storm-conf.ser and storm.jar
> should be written into the blob store instead of residing locally. We need a
> default implementation of the blob store that does essentially what nimbus
> currently does and does not need anything extra. But having an HDFS backend
> too would be great for scalability and HA.
> The supervisor should provide a way to download and manage these blobs and
> provide a working directory for the worker process with symlinks to the
> blobs. It should also allow the blobs to be updated and switch the symlink
> atomically to point to the new blob once it is downloaded.
> All of this is already done by code internal to Yahoo! we are in the process
> of getting it ready to push back to open source shortly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
