agresch commented on a change in pull request #3363:
URL: https://github.com/apache/storm/pull/3363#discussion_r546884368
##########
File path:
storm-server/src/main/java/org/apache/storm/localizer/LocallyCachedBlob.java
##########
@@ -295,6 +301,90 @@ public String getKey() {
public abstract boolean isFullyDownloaded();
+ /**
+ * Checks to see if the local blob requires update with respect to a
remote blob.
+ *
+ * @param blobStore the client blobstore
+ * @param remoteBlobstoreModTime last modification time of remote blobstore
+ * @return true of the local blob requires update, false otherwise.
+ *
+ * @throws KeyNotFoundException if the remote blob is missing
+ * @throws AuthorizationException if authorization is failed
+ */
+ boolean requiresUpdate(ClientBlobStore blobStore, long
remoteBlobstoreModTime) throws KeyNotFoundException, AuthorizationException {
+ if (!this.isUsed()) {
+ return false;
+ }
+
+ if (!this.isFullyDownloaded()) {
+ return true;
+ }
+
+ // If we are already up to date with respect to the remote blob store,
don't query
+ // the remote blobstore for the remote file. This reduces Hadoop
namenode impact of
+ // 100's of supervisors querying multiple blobs.
+ if (remoteBlobstoreModTime > 0 && this.updatedModTime ==
remoteBlobstoreModTime) {
+ LOG.debug("{} is up to date, blob updatedModTime matches remote
timestamp {}", this, remoteBlobstoreModTime);
+ return false;
+ }
+
Review comment:
Line 326 is an early out that will only occur now for HDFS blobstores.
Unsupported blobstores with -1 remoteBlobstoreModTime should act the same as
previous to this change, where they always check the local version against the
remote version of the blob.
Line 337 should work regardless of what value is passed for
remoteBlobstoreModTime.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
