Ethanlm commented on a change in pull request #3363:
URL: https://github.com/apache/storm/pull/3363#discussion_r555136615
##########
File path:
external/storm-hdfs-blobstore/src/main/java/org/apache/storm/hdfs/blobstore/HdfsBlobStore.java
##########
@@ -404,4 +404,18 @@ public void writeMetadata(String key, SettableBlobMeta
meta)
public void fullCleanup(long age) throws IOException {
hbs.fullCleanup(age);
}
+
+ public long getLastModeTime() throws IOException {
Review comment:
Nit: This should be `getLastModTime`
##########
File path:
external/storm-hdfs-blobstore/src/main/java/org/apache/storm/hdfs/blobstore/HdfsBlobStoreImpl.java
##########
@@ -312,4 +313,54 @@ public void shutdown() {
timer.cancel();
}
}
+
+ /**
+ * Get the last modification time of any blob.
+ *
+ * @return the last modification time of blobs within the blobstore.
+ * @throws IOException on any error
+ */
+ public long getLastModTime() throws IOException {
+ long modtime =
fileSystem.getFileStatus(fullPath).getModificationTime();
+ return modtime;
+ }
+
+ /**
+ * Updates the modification time of the blobstore to the current time.
+ *
+ * @throws IOException on any error
+ */
+ public void updateLastModTime() throws IOException {
+ long timestamp = Time.currentTimeMillis();
+ fileSystem.setTimes(fullPath, timestamp, timestamp);
+ LOG.debug("Updated blobstore modtime of {} to {}", fullPath,
timestamp);
+ }
+
+ /**
+ * Validates that the modification time of the blobstore is up to date
with the current existing blobs.
+ *
+ * @throws IOException on any error
+ */
+ public void validateModTime() throws IOException {
+ int currentBucket = 0;
+ long baseModTime = 0;
+ while (currentBucket < BUCKETS) {
+ String name = String.valueOf(currentBucket);
+ Path bucketDir = new Path(fullPath, name);
+
+ // only consider bucket dirs that exist with files in them
+ if (fileSystem.exists(bucketDir) &&
fileSystem.listStatus(bucketDir).length > 0) {
+ long modtime =
fileSystem.getFileStatus(bucketDir).getModificationTime();
Review comment:
Do you mean the modification timestamp of `bucketDir`?
Its modification time will not change if the content of a blob in the
directory changes. If a blob is added or deleted in the directory, the
modification time of the directory (`bucketDir`) will change. But here we
ignore any directory that has none blobs, which means the following case is not
captured: "all blobs in the directory are deleted"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
