[
https://issues.apache.org/jira/browse/STORM-876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15007614#comment-15007614
]
ASF GitHub Bot commented on STORM-876:
--------------------------------------
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/storm/pull/845#discussion_r45000490
--- Diff: storm-core/src/clj/backtype/storm/daemon/supervisor.clj ---
@@ -927,31 +1133,32 @@
first ))
(defmethod download-storm-code
- :local [conf storm-id master-code-dir supervisor download-lock]
- (let [stormroot (supervisor-stormdist-root conf storm-id)]
- (locking download-lock
- (FileUtils/copyDirectory (File. master-code-dir) (File.
stormroot))
- (let [classloader (.getContextClassLoader
(Thread/currentThread))
- resources-jar (resources-jar)
- url (.getResource classloader RESOURCES-SUBDIR)
- target-dir (str stormroot file-path-separator
RESOURCES-SUBDIR)]
- (cond
- resources-jar
- (do
- (log-message "Extracting resources from jar at "
resources-jar " to " target-dir)
- (extract-dir-from-jar resources-jar RESOURCES-SUBDIR
stormroot))
- url
- (do
- (log-message "Copying resources at " (URI. (str url)) "
to " target-dir)
- (if (= (.getProtocol url) "jar" )
- (extract-dir-from-jar (.getFile (.getJarFileURL
(.openConnection url))) RESOURCES-SUBDIR stormroot)
- (FileUtils/copyDirectory (File. (.getPath (URI. (str
url)))) (File. target-dir)))
- )
- )
- )
- )))
-
-(defmethod mk-code-distributor :local [conf] nil)
+ :local [conf storm-id master-code-dir localizer]
+ (let [tmproot (str (supervisor-tmp-dir conf) file-path-separator (uuid))
+ stormroot (supervisor-stormdist-root conf storm-id)
+ blob-store (Utils/getNimbusBlobStore conf master-code-dir nil)]
+ (try
+ (FileUtils/forceMkdir (File. tmproot))
+ (.readBlobTo blob-store (master-stormcode-key storm-id)
(FileOutputStream. (supervisor-stormcode-path tmproot)) nil)
+ (.readBlobTo blob-store (master-stormconf-key storm-id)
(FileOutputStream. (supervisor-stormconf-path tmproot)) nil)
+ (finally
+ (.shutdown blob-store)))
+ (FileUtils/moveDirectory (File. tmproot) (File. stormroot))
+ (setup-storm-code-dir conf (read-supervisor-storm-conf conf storm-id)
stormroot)
+ (let [classloader (.getContextClassLoader (Thread/currentThread))
+ resources-jar (resources-jar)
+ url (.getResource classloader RESOURCES-SUBDIR)
+ target-dir (str stormroot file-path-separator RESOURCES-SUBDIR)]
+ (cond
+ resources-jar
--- End diff --
removed it
> Dist Cache: Basic Functionality
> -------------------------------
>
> Key: STORM-876
> URL: https://issues.apache.org/jira/browse/STORM-876
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-core
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: DISTCACHE.md, DistributedCacheDesignDocument.pdf
>
>
> Basic functionality for the Dist Cache feature.
> As part of this a new API should be added to support uploading and
> downloading dist cache items. storm-core.ser, storm-conf.ser and storm.jar
> should be written into the blob store instead of residing locally. We need a
> default implementation of the blob store that does essentially what nimbus
> currently does and does not need anything extra. But having an HDFS backend
> too would be great for scalability and HA.
> The supervisor should provide a way to download and manage these blobs and
> provide a working directory for the worker process with symlinks to the
> blobs. It should also allow the blobs to be updated and switch the symlink
> atomically to point to the new blob once it is downloaded.
> All of this is already done by code internal to Yahoo! we are in the process
> of getting it ready to push back to open source shortly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
